[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:51:53 dist-test-slave-6zbq krb5kdc[4228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:51:53 dist-test-slave-6zbq krb5kdc[4228](info): set up 2 sockets
Aug 12 14:51:53 dist-test-slave-6zbq krb5kdc[4228](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 14:51:55.021008 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:51:55.128959 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:45957
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:45957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:55.209901 4250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:55.210096 4250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:55.210125 4250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:55.211236 4250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:51:55.211277 4250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:55.211294 4250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:55.211313 4250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:51:55.211328 4250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:51:55.212754 4250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:45957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:45957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:55.213013 4250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:55.213222 4250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:55.217986 4258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:55.217999 4265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:55.218030 4250 server_base.cc:1034] running on GCE node
W20260812 14:51:55.217999 4262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:55.218467 4250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:55.218748 4250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:55.219882 4250 hybrid_clock.cc:648] HybridClock initialized: now 1786546315219908 us; error 116 us; skew 500 ppm
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:55.222246 4250 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:51:55.222663 4250 webserver.cc:460] Webserver started at http://127.4.27.62:32819/ using document root <none> and password file <none>
I20260812 14:51:55.222870 4250 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:55.222918 4250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:55.223042 4250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:55.224087 4250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "6a37894825bb424d87c82a2734fb4240"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "016eea69c75f30acd4efe4a1a03c0c0f"
server_key_iv: "5560fbe6aa3929b1482a35f43d11cff4"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.224421 4250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "6a37894825bb424d87c82a2734fb4240"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "016eea69c75f30acd4efe4a1a03c0c0f"
server_key_iv: "5560fbe6aa3929b1482a35f43d11cff4"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.225602 4250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:51:55.226254 4276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.226421 4250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:51:55.226504 4250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "6a37894825bb424d87c82a2734fb4240"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "016eea69c75f30acd4efe4a1a03c0c0f"
server_key_iv: "5560fbe6aa3929b1482a35f43d11cff4"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.226562 4250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:55.237085 4250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:55.239409 4250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:55.239540 4250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:55.243204 4250 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:45957
I20260812 14:51:55.243253 4363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:45957 every 8 connection(s)
I20260812 14:51:55.243547 4250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:51:55.244386 4366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:55.246610 4366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240: Bootstrap starting.
I20260812 14:51:55.247161 4366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:55.247426 4366 log.cc:824] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:55.248054 4366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240: No bootstrap required, opened a new log
I20260812 14:51:55.249650 4366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a37894825bb424d87c82a2734fb4240" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45957 } }
I20260812 14:51:55.249809 4366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:55.249843 4366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a37894825bb424d87c82a2734fb4240, State: Initialized, Role: FOLLOWER
I20260812 14:51:55.249985 4366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a37894825bb424d87c82a2734fb4240" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45957 } }
I20260812 14:51:55.250061 4366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:51:55.250092 4366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:51:55.250144 4366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:51:55.250669 4366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a37894825bb424d87c82a2734fb4240" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45957 } }
I20260812 14:51:55.250761 4366 leader_election.cc:302] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a37894825bb424d87c82a2734fb4240; no voters:
I20260812 14:51:55.250909 4366 leader_election.cc:288] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:51:55.250943 4372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:51:55.251098 4372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [term 1 LEADER]: Becoming Leader. State: Replica: 6a37894825bb424d87c82a2734fb4240, State: Running, Role: LEADER
I20260812 14:51:55.251166 4366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:51:55.251190 4372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a37894825bb424d87c82a2734fb4240" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45957 } }
I20260812 14:51:55.251724 4374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a37894825bb424d87c82a2734fb4240. Latest consensus state: current_term: 1 leader_uuid: "6a37894825bb424d87c82a2734fb4240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a37894825bb424d87c82a2734fb4240" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45957 } } }
I20260812 14:51:55.251820 4374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [sys.catalog]: This master's current role is: LEADER
I20260812 14:51:55.252193 4383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:51:55.252193 4373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a37894825bb424d87c82a2734fb4240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a37894825bb424d87c82a2734fb4240" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45957 } } }
I20260812 14:51:55.252300 4373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240 [sys.catalog]: This master's current role is: LEADER
I20260812 14:51:55.252651 4383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:51:55.253412 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4250
I20260812 14:51:55.253511 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:51:55.254014 4383 catalog_manager.cc:1347] Generated new cluster ID: 279c112703464fb595b5f72d1b645bae
I20260812 14:51:55.254055 4383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:51:55.254268 4204 external_mini_cluster.cc:1428] Setting key 2b44c043ed751a86fec5ce8b8a162625
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:55.271625 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.255783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.4.27.62:45957)
0812 14:51:55.257806 (+ 2023us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:55.257815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:55.257842 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:51:55.257940 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:55.257945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:55.257993 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:55.258105 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:55.260254 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.261613 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.263129 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.263697 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.263789 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.267722 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:55.267726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:55.267727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:55.267738 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:55.269555 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.270066 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.270069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.270069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.270176 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.270496 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.270496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.270497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.270816 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:55.270978 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:55.271216 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 14:51:55.271405 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1979,"thread_start_us":77,"threads_started":1}
I20260812 14:51:55.273401 4383 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:51:55.273864 4383 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:51:55.282425 4383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a37894825bb424d87c82a2734fb4240: Generated new TSK 0
I20260812 14:51:55.282567 4383 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:51:55.309417 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45957
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:55.390312 4419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:55.390539 4419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:55.390561 4419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:55.391695 4419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:55.391731 4419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:55.391757 4419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:51:55.393162 4419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:55.393376 4419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:55.393591 4419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:55.398175 4439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:55.398152 4435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:55.398185 4437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:55.398361 4419 server_base.cc:1034] running on GCE node
I20260812 14:51:55.398535 4419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:55.398763 4419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:55.399900 4419 hybrid_clock.cc:648] HybridClock initialized: now 1786546315399885 us; error 28 us; skew 500 ppm
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:55.402223 4419 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:51:55.402652 4419 webserver.cc:460] Webserver started at http://127.4.27.1:33905/ using document root <none> and password file <none>
I20260812 14:51:55.402820 4419 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:55.402858 4419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:55.402945 4419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:55.403851 4419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "47b934147bfc400796a39eb8403f7655"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "ea5f47590e4af313a35f68add792d66c"
server_key_iv: "c809ab5133372204ffc655218a560b4a"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.404127 4419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "47b934147bfc400796a39eb8403f7655"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "ea5f47590e4af313a35f68add792d66c"
server_key_iv: "c809ab5133372204ffc655218a560b4a"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.405263 4419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:51:55.405858 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.405999 4419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:51:55.406076 4419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "47b934147bfc400796a39eb8403f7655"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "ea5f47590e4af313a35f68add792d66c"
server_key_iv: "c809ab5133372204ffc655218a560b4a"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.406129 4419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:55.433154 4419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:55.434900 4419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:55.435006 4419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:55.435199 4419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:51:55.435447 4419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:55.435477 4419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.435497 4419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:55.435513 4419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.440908 4419 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:35241
I20260812 14:51:55.440979 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:35241 every 8 connection(s)
I20260812 14:51:55.441267 4419 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:51:55.444808 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4419
I20260812 14:51:55.444880 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:51:55.445006 4204 external_mini_cluster.cc:1428] Setting key c0756d732460d93989754287fdb8fc46
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:55.450462 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.441949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41207 (local address 127.4.27.62:45957)
0812 14:51:55.442002 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:55.442004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:55.442186 (+ 182us) server_negotiation.cc:407] Connection header received
0812 14:51:55.442716 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:55.442718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:55.442734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:55.442785 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:55.443701 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.444203 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.445300 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.445548 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.445611 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.447779 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:55.447780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:55.447780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:55.447787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:55.449129 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.449507 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.449508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.449509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.449575 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.449795 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.449796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.449796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.449936 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:55.450007 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:55.450259 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 14:51:55.450374 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:51:55.450511 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.441898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:45957 (local address 127.4.27.1:41207)
0812 14:51:55.442064 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:55.442101 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:55.442600 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:55.442862 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:55.442867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:55.443413 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:55.443596 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:55.443601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.444289 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.444290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:55.445196 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:55.445200 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.445627 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.445628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:55.445760 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.446208 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:55.446216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:55.447689 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:55.449227 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:55.449233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:55.449235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:55.449441 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:55.449636 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:55.449643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:55.449643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:55.449731 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:55.450000 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:55.450005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:55.450132 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:51:55.450218 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:51:55.450333 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260812 14:51:55.451004 4666 heartbeater.cc:347] Connected to a master server at 127.4.27.62:45957
I20260812 14:51:55.451103 4666 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:55.451294 4666 heartbeater.cc:511] Master 127.4.27.62:45957 requested a full tablet report, sending...
I20260812 14:51:55.451814 4300 ts_manager.cc:194] Registered new tserver with Master: 47b934147bfc400796a39eb8403f7655 (127.4.27.1:35241)
I20260812 14:51:55.452466 4300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:41207
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:51:55.469877 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45957
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:55.549824 4671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:55.550014 4671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:55.550045 4671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:55.551236 4671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:55.551281 4671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:55.551317 4671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:51:55.552838 4671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:55.553081 4671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:55.553296 4671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:55.558064 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:55.558049 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:55.558154 4671 server_base.cc:1034] running on GCE node
W20260812 14:51:55.558048 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:55.558465 4671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:55.558707 4671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:55.559844 4671 hybrid_clock.cc:648] HybridClock initialized: now 1786546315559829 us; error 31 us; skew 500 ppm
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:55.562110 4671 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:51:55.562552 4671 webserver.cc:460] Webserver started at http://127.4.27.2:41311/ using document root <none> and password file <none>
I20260812 14:51:55.562741 4671 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:55.562788 4671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:55.562908 4671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:55.563884 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "a2c41e50217c4076b3ca76c3c33ba811"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "f4adba92ca23e34df6f17889350ca639"
server_key_iv: "e34ed3a6df2af773e05f846223faa671"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.564221 4671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "a2c41e50217c4076b3ca76c3c33ba811"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "f4adba92ca23e34df6f17889350ca639"
server_key_iv: "e34ed3a6df2af773e05f846223faa671"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.565409 4671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:51:55.566043 4698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.566201 4671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:51:55.566285 4671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "a2c41e50217c4076b3ca76c3c33ba811"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "f4adba92ca23e34df6f17889350ca639"
server_key_iv: "e34ed3a6df2af773e05f846223faa671"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.566371 4671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:55.578683 4671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:55.580368 4671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:55.580495 4671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:55.580695 4671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:51:55.580971 4671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:55.581002 4671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.581029 4671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:55.581048 4671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.587004 4671 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:38949
I20260812 14:51:55.587042 4905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:38949 every 8 connection(s)
I20260812 14:51:55.587334 4671 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:55.594164 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4671
I20260812 14:51:55.594249 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:51:55.594452 4204 external_mini_cluster.cc:1428] Setting key de8790b8e009c967dcdb52a31f268c13
I20260812 14:51:55.595563 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.587969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:37331 (local address 127.4.27.62:45957)
0812 14:51:55.588031 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:55.588033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:55.588194 (+ 161us) server_negotiation.cc:407] Connection header received
0812 14:51:55.588666 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:55.588667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:55.588677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:55.588718 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:55.589565 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.590053 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.590998 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.591176 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.591224 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.593056 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:55.593057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:55.593057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:55.593064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:55.594239 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.594690 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.594691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.594691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.594752 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.594974 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.594975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.594976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.595082 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:55.595141 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:55.595438 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 14:51:55.595503 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:51:55.595762 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.587920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:45957 (local address 127.4.27.2:37331)
0812 14:51:55.588073 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:55.588112 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:55.588568 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:55.588781 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:55.588784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:55.589319 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:55.589500 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:55.589504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.590113 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.590113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:55.590935 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:55.590937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.591233 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.591234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:55.591325 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.591708 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:55.591715 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:55.592965 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:55.594306 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:55.594313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:55.594316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:55.594600 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:55.594809 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:55.594810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:55.594811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:55.594918 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:55.595163 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:55.595168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:55.595307 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 14:51:55.595423 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 14:51:55.595568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260812 14:51:55.596138 4908 heartbeater.cc:347] Connected to a master server at 127.4.27.62:45957
I20260812 14:51:55.596227 4908 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:55.596388 4908 heartbeater.cc:511] Master 127.4.27.62:45957 requested a full tablet report, sending...
I20260812 14:51:55.596743 4300 ts_manager.cc:194] Registered new tserver with Master: a2c41e50217c4076b3ca76c3c33ba811 (127.4.27.2:38949)
I20260812 14:51:55.597090 4300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:37331
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:51:55.617650 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45957
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:55.696838 4918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:55.697028 4918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:55.697059 4918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:55.698148 4918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:55.698189 4918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:55.698223 4918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:51:55.699649 4918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:55.699872 4918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:55.700064 4918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:55.704593 4930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:55.704605 4925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:55.704622 4923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:55.704844 4918 server_base.cc:1034] running on GCE node
I20260812 14:51:55.705036 4918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:55.705219 4918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:55.706354 4918 hybrid_clock.cc:648] HybridClock initialized: now 1786546315706326 us; error 28 us; skew 500 ppm
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:55.708544 4918 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:51:55.708986 4918 webserver.cc:460] Webserver started at http://127.4.27.3:36127/ using document root <none> and password file <none>
I20260812 14:51:55.709179 4918 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:55.709220 4918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:55.709334 4918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:55.710325 4918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "b6740a5633e643fa9c11e3899e7182d4"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "2ffa894eb0043e75424a944ba419d738"
server_key_iv: "15976433e0a4e97ccfbfdd280aa05a7d"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.710695 4918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "b6740a5633e643fa9c11e3899e7182d4"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "2ffa894eb0043e75424a944ba419d738"
server_key_iv: "15976433e0a4e97ccfbfdd280aa05a7d"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.711877 4918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:51:55.712491 4941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.712625 4918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:51:55.712719 4918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "b6740a5633e643fa9c11e3899e7182d4"
format_stamp: "Formatted at 2026-08-12 14:51:55 on dist-test-slave-6zbq"
server_key: "2ffa894eb0043e75424a944ba419d738"
server_key_iv: "15976433e0a4e97ccfbfdd280aa05a7d"
server_key_version: "encryptionkey@0"
I20260812 14:51:55.712780 4918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:55.733939 4918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:55.735623 4918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:55.735751 4918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:55.735934 4918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:51:55.736207 4918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:55.736250 4918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.736277 4918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:55.736300 4918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:55.741796 4918 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:44599
I20260812 14:51:55.741850 5153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:44599 every 8 connection(s)
I20260812 14:51:55.742095 4918 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:51:55.742700 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4918
I20260812 14:51:55.742786 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:51:55.742945 4204 external_mini_cluster.cc:1428] Setting key 05d0a3649a2e145f6860be618e33fd12
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:55.750031 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.743028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:51955 (local address 127.4.27.62:45957)
0812 14:51:55.743073 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:55.743075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:55.743087 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:51:55.743613 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:55.743614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:55.743625 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:55.743673 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:55.744598 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.745083 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.745986 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.746180 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.746235 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.748043 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:55.748044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:55.748045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:55.748051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:55.749061 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.749385 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.749386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.749386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.749428 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.749594 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.749595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.749595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.749686 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:55.749729 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:55.749918 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:51:55.749973 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:51:55.750192 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.742810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:45957 (local address 127.4.27.3:51955)
0812 14:51:55.742967 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:55.743013 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:55.743498 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:55.743737 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:55.743740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:55.744326 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:55.744507 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:55.744511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.745148 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.745148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:55.745899 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:55.745900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.746247 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.746247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:55.746332 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.746710 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:55.746718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:55.747974 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:55.749123 (+ 1149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:55.749127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:55.749129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:55.749329 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:55.749471 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:55.749471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:55.749471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:55.749555 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:55.749727 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:55.749729 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:55.749835 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 14:51:55.749907 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:51:55.750014 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260812 14:51:55.750589 5155 heartbeater.cc:347] Connected to a master server at 127.4.27.62:45957
I20260812 14:51:55.750681 5155 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:55.750851 5155 heartbeater.cc:511] Master 127.4.27.62:45957 requested a full tablet report, sending...
I20260812 14:51:55.751209 4300 ts_manager.cc:194] Registered new tserver with Master: b6740a5633e643fa9c11e3899e7182d4 (127.4.27.3:44599)
I20260812 14:51:55.751546 4300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:51955
I20260812 14:51:55.754813 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:51:55.775926 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.769405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.4.27.1:35241)
0812 14:51:55.769551 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:55.769555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:55.769563 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:51:55.769584 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:55.769586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:55.769599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:55.769661 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:55.770176 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.770910 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.771576 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.771863 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.771910 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.773713 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:55.773723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:55.773723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:55.773730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:55.775111 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.775376 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.775379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.775379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.775428 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.775544 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.775544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.775544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.775651 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:55.775740 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:55.775796 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 14:51:55.775854 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
W20260812 14:51:55.776247 4505 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37744
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:51:55.795104 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:55.789573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.4.27.1:35241)
0812 14:51:55.789622 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:55.789623 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:55.789685 (+ 62us) server_negotiation.cc:407] Connection header received
0812 14:51:55.789719 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:55.789720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:55.789728 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:55.789773 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:55.790222 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.790696 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.791318 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:55.791493 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:55.791532 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:55.793310 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:55.793311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:55.793311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:55.793317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:55.794315 (+ 998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.794617 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.794618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.794618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.794661 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:55.794795 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:55.794796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:55.794796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:55.794866 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:55.794941 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:55.794996 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 14:51:55.795050 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:51:55.795571 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4419
I20260812 14:51:55.807878 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4671
I20260812 14:51:55.829190 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4918
I20260812 14:51:55.837186 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4250
2026-08-12T14:51:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2039 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[5182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[5182](info): set up 2 sockets
Aug 12 14:51:55 dist-test-slave-6zbq krb5kdc[5182](info): commencing operation
krb5kdc: starting...
W20260812 14:51:57.010834 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:51:57.118551 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46115
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:57.197597 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:57.197791 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:57.197813 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:57.198956 5210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:51:57.198993 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:57.199007 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:57.199028 5210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:51:57.199045 5210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:51:57.200469 5210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:57.200690 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:57.200878 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:57.205303 5222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.205303 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.205308 5227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:57.205542 5210 server_base.cc:1034] running on GCE node
I20260812 14:51:57.205762 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:57.206002 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:57.207132 5210 hybrid_clock.cc:648] HybridClock initialized: now 1786546317207119 us; error 29 us; skew 500 ppm
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:57.209354 5210 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:51:57.209762 5210 webserver.cc:460] Webserver started at http://127.4.27.62:39489/ using document root <none> and password file <none>
I20260812 14:51:57.209947 5210 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:57.209991 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:57.210114 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:57.211110 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "7aead8b648a64c788a47f9f2d2699e3e"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "bab21188ffa374f1630e1e3625fad4a0"
server_key_iv: "dee47d0f5149717beb02f558888a6894"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.211426 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "7aead8b648a64c788a47f9f2d2699e3e"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "bab21188ffa374f1630e1e3625fad4a0"
server_key_iv: "dee47d0f5149717beb02f558888a6894"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.212626 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:51:57.213251 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.213394 5210 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:51:57.213477 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "7aead8b648a64c788a47f9f2d2699e3e"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "bab21188ffa374f1630e1e3625fad4a0"
server_key_iv: "dee47d0f5149717beb02f558888a6894"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.213536 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:57.241420 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:57.243168 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:57.243288 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:57.247152 5210 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46115
I20260812 14:51:57.247292 5364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46115 every 8 connection(s)
I20260812 14:51:57.247630 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:51:57.249177 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:57.252665 5365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e: Bootstrap starting.
I20260812 14:51:57.252944 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5210
I20260812 14:51:57.253033 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:51:57.253201 4204 external_mini_cluster.cc:1428] Setting key 90983ba2d5895edb4924341c0fd0fe8a
I20260812 14:51:57.253724 5365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:57.254117 5365 log.cc:824] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:57.254916 5365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e: No bootstrap required, opened a new log
I20260812 14:51:57.257145 5365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aead8b648a64c788a47f9f2d2699e3e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46115 } }
I20260812 14:51:57.257303 5365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:57.257328 5365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aead8b648a64c788a47f9f2d2699e3e, State: Initialized, Role: FOLLOWER
I20260812 14:51:57.257457 5365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aead8b648a64c788a47f9f2d2699e3e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46115 } }
I20260812 14:51:57.257529 5365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:51:57.257555 5365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:51:57.257601 5365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:51:57.258275 5365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aead8b648a64c788a47f9f2d2699e3e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46115 } }
I20260812 14:51:57.258405 5365 leader_election.cc:302] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aead8b648a64c788a47f9f2d2699e3e; no voters:
I20260812 14:51:57.258580 5365 leader_election.cc:288] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:51:57.258630 5375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:51:57.258745 5375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [term 1 LEADER]: Becoming Leader. State: Replica: 7aead8b648a64c788a47f9f2d2699e3e, State: Running, Role: LEADER
I20260812 14:51:57.258838 5375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aead8b648a64c788a47f9f2d2699e3e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46115 } }
I20260812 14:51:57.258867 5365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:51:57.259300 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aead8b648a64c788a47f9f2d2699e3e. Latest consensus state: current_term: 1 leader_uuid: "7aead8b648a64c788a47f9f2d2699e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aead8b648a64c788a47f9f2d2699e3e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46115 } } }
I20260812 14:51:57.259380 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [sys.catalog]: This master's current role is: LEADER
I20260812 14:51:57.259527 5377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aead8b648a64c788a47f9f2d2699e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aead8b648a64c788a47f9f2d2699e3e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46115 } } }
I20260812 14:51:57.259580 5377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e [sys.catalog]: This master's current role is: LEADER
I20260812 14:51:57.259845 5380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:57.260330 5380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:51:57.262094 5380 catalog_manager.cc:1347] Generated new cluster ID: 73f8ff98f6654b708096369842b49e06
I20260812 14:51:57.262162 5380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:51:57.264568 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.253854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.4.27.62:46115)
0812 14:51:57.254067 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.254075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.254105 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:51:57.254850 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.254859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.254926 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.255075 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.255674 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.256562 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.257274 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.257846 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.257945 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.260858 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.260863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.260863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.260876 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.262815 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.263168 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.263171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.263171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.263265 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.263437 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.263438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.263438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.263708 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.264172 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.264247 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.264370 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260812 14:51:57.279026 5380 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:51:57.279528 5380 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:51:57.282505 5380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aead8b648a64c788a47f9f2d2699e3e: Generated new TSK 0
I20260812 14:51:57.282662 5380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:51:57.313239 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46115
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:57.397025 5417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:57.397223 5417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:57.397251 5417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:57.398389 5417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:57.398425 5417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:57.398454 5417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:51:57.399919 5417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:57.400142 5417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:57.400399 5417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:57.405184 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.405172 5430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.405205 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:57.405301 5417 server_base.cc:1034] running on GCE node
I20260812 14:51:57.405599 5417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:57.405843 5417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:57.406963 5417 hybrid_clock.cc:648] HybridClock initialized: now 1786546317406951 us; error 31 us; skew 500 ppm
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:57.409258 5417 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:51:57.409641 5417 webserver.cc:460] Webserver started at http://127.4.27.1:34131/ using document root <none> and password file <none>
I20260812 14:51:57.409840 5417 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:57.409884 5417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:57.410007 5417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:57.411016 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "37fad715cd3244bc9558bd3788a5dae8"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "0042043db0f97e25466451b944ae4090"
server_key_iv: "783c3eeb77bed14f357fb44d31dcfcdb"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.411332 5417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "37fad715cd3244bc9558bd3788a5dae8"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "0042043db0f97e25466451b944ae4090"
server_key_iv: "783c3eeb77bed14f357fb44d31dcfcdb"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.412550 5417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:51:57.413198 5447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.413365 5417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:51:57.413447 5417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "37fad715cd3244bc9558bd3788a5dae8"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "0042043db0f97e25466451b944ae4090"
server_key_iv: "783c3eeb77bed14f357fb44d31dcfcdb"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.413506 5417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:57.439352 5417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:57.440965 5417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:57.441104 5417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:57.441300 5417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:51:57.441578 5417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:57.441620 5417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.441648 5417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:57.441664 5417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.447064 5417 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37301
I20260812 14:51:57.447124 5642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37301 every 8 connection(s)
I20260812 14:51:57.447386 5417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:51:57.447956 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5417
I20260812 14:51:57.448043 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:51:57.448195 4204 external_mini_cluster.cc:1428] Setting key 2a682e179ad3540f6c4e7b936e846aba
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:57.456557 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.448042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:57713 (local address 127.4.27.62:46115)
0812 14:51:57.448106 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.448108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.448407 (+ 299us) server_negotiation.cc:407] Connection header received
0812 14:51:57.449558 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.449559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.449572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.449635 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.450197 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.450791 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.451723 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.451982 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.452048 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.453964 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.453965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.453966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.453973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.455384 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.455735 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.455736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.455737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.455798 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.455991 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.455992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.455992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.456118 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.456189 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.456387 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.456469 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:51:57.456599 5648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.447972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46115 (local address 127.4.27.1:57713)
0812 14:51:57.448273 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.448313 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.449449 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.449696 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.449699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.449928 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:57.450101 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.450105 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.450877 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.450879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.451642 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.451645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.452060 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.452062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.452195 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.452619 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:57.452627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:57.453879 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:57.455456 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:57.455462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:57.455464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:57.455667 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:57.455856 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:57.455856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:57.455856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:57.455938 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:57.456176 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:57.456178 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:57.456277 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.456348 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.456455 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
I20260812 14:51:57.457125 5644 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46115
I20260812 14:51:57.457231 5644 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:57.457387 5644 heartbeater.cc:511] Master 127.4.27.62:46115 requested a full tablet report, sending...
I20260812 14:51:57.457827 5282 ts_manager.cc:194] Registered new tserver with Master: 37fad715cd3244bc9558bd3788a5dae8 (127.4.27.1:37301)
I20260812 14:51:57.458417 5282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:57713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:51:57.470851 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46115
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:57.548861 5656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:57.549037 5656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:57.549068 5656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:57.550192 5656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:57.550231 5656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:57.550266 5656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:51:57.551741 5656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:57.551970 5656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:57.552173 5656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:51:57.556784 5656 server_base.cc:1034] running on GCE node
W20260812 14:51:57.556733 5676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.556739 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.556738 5664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:57.557188 5656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:57.557408 5656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:57.558539 5656 hybrid_clock.cc:648] HybridClock initialized: now 1786546317558527 us; error 29 us; skew 500 ppm
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:57.560967 5656 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:51:57.561348 5656 webserver.cc:460] Webserver started at http://127.4.27.2:35601/ using document root <none> and password file <none>
I20260812 14:51:57.561546 5656 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:57.561589 5656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:57.561697 5656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:57.562680 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "619e22d3590e45ab86b71cbe0e26e2a8"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "5fa3274010383a0177fbcb79c8407768"
server_key_iv: "0d9f9ab926615e6807c2e2a8253371bd"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.562973 5656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "619e22d3590e45ab86b71cbe0e26e2a8"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "5fa3274010383a0177fbcb79c8407768"
server_key_iv: "0d9f9ab926615e6807c2e2a8253371bd"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.564090 5656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:51:57.564772 5689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.564917 5656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:51:57.564994 5656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "619e22d3590e45ab86b71cbe0e26e2a8"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "5fa3274010383a0177fbcb79c8407768"
server_key_iv: "0d9f9ab926615e6807c2e2a8253371bd"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.565055 5656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:57.576495 5656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:57.578480 5656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:57.578603 5656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:57.578791 5656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:51:57.579063 5656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:57.579095 5656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.579116 5656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:57.579139 5656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.584532 5656 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:42591
I20260812 14:51:57.584589 5875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:42591 every 8 connection(s)
I20260812 14:51:57.584872 5656 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:57.592701 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.585513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:33725 (local address 127.4.27.62:46115)
0812 14:51:57.585576 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.585577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.585751 (+ 174us) server_negotiation.cc:407] Connection header received
0812 14:51:57.586692 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.586693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.586703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.586755 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.587281 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.587769 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.588680 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.588851 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.588892 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.590690 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.590691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.590691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.590698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.591728 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.592047 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.592047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.592048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.592089 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.592254 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.592254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.592255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.592345 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.592387 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.592577 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.592632 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:51:57.592844 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.585475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46115 (local address 127.4.27.2:33725)
0812 14:51:57.585632 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.585672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.586608 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.586815 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.586819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.587045 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:57.587220 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.587224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.587829 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.587829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.588621 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.588623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.588903 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.588903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.588999 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.589363 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:57.589371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:57.590606 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:57.591786 (+ 1180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:57.591792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:57.591793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:57.591993 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:57.592133 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:57.592133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:57.592133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:57.592215 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:57.592385 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:57.592387 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:57.592487 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.592559 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.592666 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260812 14:51:57.593255 5876 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46115
I20260812 14:51:57.593349 5876 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:57.593506 5876 heartbeater.cc:511] Master 127.4.27.62:46115 requested a full tablet report, sending...
I20260812 14:51:57.593847 5282 ts_manager.cc:194] Registered new tserver with Master: 619e22d3590e45ab86b71cbe0e26e2a8 (127.4.27.2:42591)
I20260812 14:51:57.594179 5282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:33725
I20260812 14:51:57.594754 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5656
I20260812 14:51:57.594844 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:51:57.594995 4204 external_mini_cluster.cc:1428] Setting key 75890d6a3a12102b5dd1e153e26a5d42
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:51:57.618759 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46115
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:57.699260 5888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:57.699440 5888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:57.699461 5888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:57.700520 5888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:57.700553 5888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:57.700578 5888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:51:57.702013 5888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:57.702180 5888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:57.702373 5888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:57.706885 5905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.706889 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:57.706883 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:57.707062 5888 server_base.cc:1034] running on GCE node
I20260812 14:51:57.707285 5888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:57.707473 5888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:57.708602 5888 hybrid_clock.cc:648] HybridClock initialized: now 1786546317708595 us; error 27 us; skew 500 ppm
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:57.710878 5888 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:51:57.711272 5888 webserver.cc:460] Webserver started at http://127.4.27.3:42603/ using document root <none> and password file <none>
I20260812 14:51:57.711476 5888 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:57.711519 5888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:57.711642 5888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:57.712620 5888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "13fb07a089744b47822e8a214b8a63b5"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "45b1bf9d7ca5e432b6da3cc56ea0d6b5"
server_key_iv: "ddff3cad75cef6e5176e79bcde823baf"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.712937 5888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "13fb07a089744b47822e8a214b8a63b5"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "45b1bf9d7ca5e432b6da3cc56ea0d6b5"
server_key_iv: "ddff3cad75cef6e5176e79bcde823baf"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.714152 5888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:51:57.714841 5919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.714980 5888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.715055 5888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "13fb07a089744b47822e8a214b8a63b5"
format_stamp: "Formatted at 2026-08-12 14:51:57 on dist-test-slave-6zbq"
server_key: "45b1bf9d7ca5e432b6da3cc56ea0d6b5"
server_key_iv: "ddff3cad75cef6e5176e79bcde823baf"
server_key_version: "encryptionkey@0"
I20260812 14:51:57.715121 5888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:57.721901 5888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:57.723599 5888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:57.723708 5888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:57.723901 5888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:51:57.724175 5888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:57.724207 5888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.724226 5888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:57.724249 5888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:57.729983 5888 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:46337
I20260812 14:51:57.730027 6109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:46337 every 8 connection(s)
I20260812 14:51:57.730294 5888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:51:57.733983 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5888
I20260812 14:51:57.734084 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:51:57.734254 4204 external_mini_cluster.cc:1428] Setting key 6f9b95b7568fce189cf016ef448afc9f
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:57.738941 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.730978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:37849 (local address 127.4.27.62:46115)
0812 14:51:57.731038 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.731040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.731245 (+ 205us) server_negotiation.cc:407] Connection header received
0812 14:51:57.732090 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.732091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.732100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.732154 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.732678 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.733165 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.734224 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.734493 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.734554 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.736548 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.736549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.736549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.736556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.737669 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.738042 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.738043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.738044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.738100 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.738324 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.738325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.738325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.738463 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.738521 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.738774 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.738856 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:51:57.739084 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.730932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46115 (local address 127.4.27.3:37849)
0812 14:51:57.731133 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.731172 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.732011 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.732203 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.732206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.732439 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:57.732616 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.732620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.733223 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.733224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.734147 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.734150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.734566 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.734567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.734709 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.735167 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:57.735177 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:57.736453 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:57.737756 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:57.737762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:57.737764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:57.737973 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:57.738168 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:57.738169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:57.738169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:57.738265 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:57.738520 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:57.738526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:57.738655 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.738743 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.738885 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260812 14:51:57.739459 6110 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46115
I20260812 14:51:57.739544 6110 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:57.739710 6110 heartbeater.cc:511] Master 127.4.27.62:46115 requested a full tablet report, sending...
I20260812 14:51:57.740031 5282 ts_manager.cc:194] Registered new tserver with Master: 13fb07a089744b47822e8a214b8a63b5 (127.4.27.3:46337)
I20260812 14:51:57.740371 5282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:37849
I20260812 14:51:57.745855 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:51:57.752144 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.747294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.4.27.62:46115)
0812 14:51:57.747351 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.747352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.747440 (+ 88us) server_negotiation.cc:407] Connection header received
0812 14:51:57.747478 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.747479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.747487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.747534 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.748006 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.748453 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.749081 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.749242 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.749284 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.750181 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.750182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.750182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.750188 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.751354 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.751602 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.751603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.751603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.751646 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.751762 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.751769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.751770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.751882 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.751950 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.752030 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.752090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:51:57.754310 5282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:51:57.755813 5282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:51:57.762092 6122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.757881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37301 (local address 127.0.0.1:50448)
0812 14:51:57.758412 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.758420 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.758461 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.758629 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.758630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.758632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:57.758680 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.758681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.760014 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.760015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.761077 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.761079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.761802 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.761803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.761889 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.761903 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.761969 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.762021 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":41,"threads_started":1}
I20260812 14:51:57.762107 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.758075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.4.27.1:37301)
0812 14:51:57.758222 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.758226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.758473 (+ 247us) server_negotiation.cc:407] Connection header received
0812 14:51:57.758520 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.758523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.758540 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.758631 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:57.758742 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.759923 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.761180 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.761727 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.761798 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.761857 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.761991 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.762036 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260812 14:51:57.762293 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.757676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:42591 (local address 127.0.0.1:38530)
0812 14:51:57.758256 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.758273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.758320 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.758682 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.758683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.758685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:57.758725 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.758726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.760069 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.760070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.761000 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.761004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.762051 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.762053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.762122 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.762127 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.762167 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.762211 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":3,"spinlock_wait_cycles":896,"thread_start_us":67,"threads_started":1}
I20260812 14:51:57.762686 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.757676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:46337 (local address 127.0.0.1:36610)
0812 14:51:57.758072 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.758102 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.758167 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.758381 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.758385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.758387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:57.758447 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.758449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.759579 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.759580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.760834 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.760836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.762477 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.762478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.762540 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.762543 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.762594 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.762629 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":52,"threads_started":1}
I20260812 14:51:57.762696 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.757830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.4.27.2:42591)
0812 14:51:57.758002 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.758006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.758531 (+ 525us) server_negotiation.cc:407] Connection header received
0812 14:51:57.758576 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.758578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.758592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.758793 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:57.758870 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.759994 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.761075 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.761811 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.762470 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.762538 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.762563 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.762605 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260812 14:51:57.762713 5533 tablet_service.cc:1467] Processing CreateTablet for tablet 5806d6bf31394e508404336fd3562f3c (DEFAULT_TABLE table=test-table [id=499054402c0147079287e99911fcfcf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:51:57.762952 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.757752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.4.27.3:46337)
0812 14:51:57.757910 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.757915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.758170 (+ 255us) server_negotiation.cc:407] Connection header received
0812 14:51:57.758235 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.758237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.758252 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.758325 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:57.758517 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.759476 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.761183 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.761830 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.761963 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.762102 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.762848 (+ 746us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.762888 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260812 14:51:57.763082 5533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5806d6bf31394e508404336fd3562f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:57.763535 5999 tablet_service.cc:1467] Processing CreateTablet for tablet 5806d6bf31394e508404336fd3562f3c (DEFAULT_TABLE table=test-table [id=499054402c0147079287e99911fcfcf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:51:57.763845 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5806d6bf31394e508404336fd3562f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:57.766317 6143 tablet_bootstrap.cc:492] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8: Bootstrap starting.
I20260812 14:51:57.766731 5760 tablet_service.cc:1467] Processing CreateTablet for tablet 5806d6bf31394e508404336fd3562f3c (DEFAULT_TABLE table=test-table [id=499054402c0147079287e99911fcfcf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:51:57.767043 6143 tablet_bootstrap.cc:654] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:57.767066 5760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5806d6bf31394e508404336fd3562f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:57.767251 6144 tablet_bootstrap.cc:492] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5: Bootstrap starting.
I20260812 14:51:57.767627 6143 log.cc:824] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:57.767951 6144 tablet_bootstrap.cc:654] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:57.768432 6143 tablet_bootstrap.cc:492] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8: No bootstrap required, opened a new log
I20260812 14:51:57.768504 6143 ts_tablet_manager.cc:1397] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:51:57.768501 6144 log.cc:824] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:57.769249 6144 tablet_bootstrap.cc:492] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5: No bootstrap required, opened a new log
I20260812 14:51:57.769316 6144 ts_tablet_manager.cc:1397] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:51:57.769891 6147 tablet_bootstrap.cc:492] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: Bootstrap starting.
I20260812 14:51:57.770489 6147 tablet_bootstrap.cc:654] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:57.770399 6143 raft_consensus.cc:348] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.770566 6143 raft_consensus.cc:374] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:57.770608 6143 raft_consensus.cc:729] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fad715cd3244bc9558bd3788a5dae8, State: Initialized, Role: FOLLOWER
I20260812 14:51:57.770717 6143 consensus_queue.cc:260] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.770974 6143 ts_tablet_manager.cc:1428] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 14:51:57.771009 6147 log.cc:824] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:57.771090 5644 heartbeater.cc:503] Master 127.4.27.62:46115 was elected leader, sending a full tablet report...
I20260812 14:51:57.771159 6144 raft_consensus.cc:348] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.771317 6144 raft_consensus.cc:374] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:57.771359 6144 raft_consensus.cc:729] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13fb07a089744b47822e8a214b8a63b5, State: Initialized, Role: FOLLOWER
I20260812 14:51:57.771462 6144 consensus_queue.cc:260] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.771727 6144 ts_tablet_manager.cc:1428] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:51:57.771785 6110 heartbeater.cc:503] Master 127.4.27.62:46115 was elected leader, sending a full tablet report...
I20260812 14:51:57.771806 6147 tablet_bootstrap.cc:492] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: No bootstrap required, opened a new log
I20260812 14:51:57.771859 6147 ts_tablet_manager.cc:1397] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:51:57.773454 6147 raft_consensus.cc:348] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.773583 6147 raft_consensus.cc:374] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:57.773604 6147 raft_consensus.cc:729] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619e22d3590e45ab86b71cbe0e26e2a8, State: Initialized, Role: FOLLOWER
I20260812 14:51:57.773705 6147 consensus_queue.cc:260] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.773938 6147 ts_tablet_manager.cc:1428] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:51:57.773967 5876 heartbeater.cc:503] Master 127.4.27.62:46115 was elected leader, sending a full tablet report...
I20260812 14:51:57.786960 6155 raft_consensus.cc:482] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:51:57.787034 6155 raft_consensus.cc:504] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.787303 6155 leader_election.cc:288] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13fb07a089744b47822e8a214b8a63b5 (127.4.27.3:46337), 37fad715cd3244bc9558bd3788a5dae8 (127.4.27.1:37301)
I20260812 14:51:57.789821 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.787425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:46337 (local address 127.4.27.2:38093)
0812 14:51:57.787471 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.787481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.787518 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.787675 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.787676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.787677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:57.787738 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.787740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.788324 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.788325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.789106 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.789109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.789597 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.789598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.789676 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.789679 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.789722 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.789762 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:51:57.789819 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.787466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:38093 (local address 127.4.27.3:46337)
0812 14:51:57.787501 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.787502 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.787511 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:51:57.787580 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.787581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.787591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.787668 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:57.787795 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.788258 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.789185 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.789530 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.789586 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.789595 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.789733 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.789762 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:51:57.789919 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.787518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:44945 (local address 127.4.27.1:37301)
0812 14:51:57.787555 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.787557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.787650 (+ 93us) server_negotiation.cc:407] Connection header received
0812 14:51:57.787692 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.787693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.787702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.787757 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:57.787877 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.788329 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.789280 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.789631 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.789689 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.789697 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.789834 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.789864 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:51:57.789932 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.787435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37301 (local address 127.4.27.2:44945)
0812 14:51:57.787585 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:57.787594 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:57.787648 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:57.787763 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:57.787763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:57.787765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:57.787827 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.787829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.788412 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.788413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.789200 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:57.789203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.789701 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.789702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:57.789776 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.789779 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:51:57.789822 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:51:57.789858 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260812 14:51:57.790092 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5806d6bf31394e508404336fd3562f3c" candidate_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fb07a089744b47822e8a214b8a63b5" is_pre_election: true
I20260812 14:51:57.790201 6031 raft_consensus.cc:2454] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619e22d3590e45ab86b71cbe0e26e2a8 in term 0.
I20260812 14:51:57.790194 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5806d6bf31394e508404336fd3562f3c" candidate_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fad715cd3244bc9558bd3788a5dae8" is_pre_election: true
I20260812 14:51:57.790285 5564 raft_consensus.cc:2454] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 619e22d3590e45ab86b71cbe0e26e2a8 in term 0.
I20260812 14:51:57.790426 5696 leader_election.cc:302] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fb07a089744b47822e8a214b8a63b5, 619e22d3590e45ab86b71cbe0e26e2a8; no voters:
I20260812 14:51:57.790529 6155 raft_consensus.cc:2781] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:51:57.790581 6155 raft_consensus.cc:482] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:51:57.790603 6155 raft_consensus.cc:3037] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:51:57.791162 6155 raft_consensus.cc:504] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.791275 6155 leader_election.cc:288] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [CANDIDATE]: Term 1 election: Requested vote from peers 13fb07a089744b47822e8a214b8a63b5 (127.4.27.3:46337), 37fad715cd3244bc9558bd3788a5dae8 (127.4.27.1:37301)
I20260812 14:51:57.791425 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5806d6bf31394e508404336fd3562f3c" candidate_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13fb07a089744b47822e8a214b8a63b5"
I20260812 14:51:57.791421 5564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5806d6bf31394e508404336fd3562f3c" candidate_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fad715cd3244bc9558bd3788a5dae8"
I20260812 14:51:57.791492 6031 raft_consensus.cc:3037] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:51:57.791492 5564 raft_consensus.cc:3037] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:51:57.792205 5564 raft_consensus.cc:2454] T 5806d6bf31394e508404336fd3562f3c P 37fad715cd3244bc9558bd3788a5dae8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619e22d3590e45ab86b71cbe0e26e2a8 in term 1.
I20260812 14:51:57.792205 6031 raft_consensus.cc:2454] T 5806d6bf31394e508404336fd3562f3c P 13fb07a089744b47822e8a214b8a63b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 619e22d3590e45ab86b71cbe0e26e2a8 in term 1.
I20260812 14:51:57.792356 5696 leader_election.cc:302] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13fb07a089744b47822e8a214b8a63b5, 619e22d3590e45ab86b71cbe0e26e2a8; no voters:
I20260812 14:51:57.792488 6155 raft_consensus.cc:2781] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:51:57.792632 6155 raft_consensus.cc:686] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [term 1 LEADER]: Becoming Leader. State: Replica: 619e22d3590e45ab86b71cbe0e26e2a8, State: Running, Role: LEADER
I20260812 14:51:57.792708 6155 consensus_queue.cc:237] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } }
I20260812 14:51:57.793368 5280 catalog_manager.cc:5495] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 619e22d3590e45ab86b71cbe0e26e2a8 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13fb07a089744b47822e8a214b8a63b5" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619e22d3590e45ab86b71cbe0e26e2a8" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37fad715cd3244bc9558bd3788a5dae8" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37301 } health_report { overall_health: UNKNOWN } } }
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:51:57.803274 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.796879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.4.27.1:37301)
0812 14:51:57.796939 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.796941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.797002 (+ 61us) server_negotiation.cc:407] Connection header received
0812 14:51:57.797051 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.797052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.797059 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.797105 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.797559 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.797962 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.798656 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.798832 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.798876 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.800853 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.800861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.800861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.800867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.802261 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.802589 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.802592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.802592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.802645 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.802810 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.802811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.802811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.802926 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.803044 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.803129 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.803197 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:51:57.824733 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.819187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.4.27.1:37301)
0812 14:51:57.819235 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.819237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.819316 (+ 79us) server_negotiation.cc:407] Connection header received
0812 14:51:57.819359 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.819360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.819367 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.819409 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.819853 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.820273 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.820903 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.821066 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.821113 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.822998 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.822999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.823000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.823006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.823977 (+ 971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.824241 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.824241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.824242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.824282 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.824397 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.824398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.824398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.824465 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.824520 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.824626 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.824679 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260812 14:51:57.835366 5877 tablet.cc:2369] T 5806d6bf31394e508404336fd3562f3c P 619e22d3590e45ab86b71cbe0e26e2a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[5182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:51:57.844238 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:57.838608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.4.27.1:37301)
0812 14:51:57.838653 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:57.838655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:57.838744 (+ 89us) server_negotiation.cc:407] Connection header received
0812 14:51:57.838802 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:57.838802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:57.838811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:57.838851 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:57.839318 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.839739 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.840387 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:57.840561 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:57.840611 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:57.842554 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:57.842555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:57.842555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:57.842561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:57.843510 (+ 949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.843778 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.843778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.843779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.843817 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:57.843932 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:57.843932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:57.843932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:57.844000 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:57.844056 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:57.844139 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:51:57.844186 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260812 14:51:57.844602 5500 tablet_service.cc:2908] Rejecting scan request for tablet 5806d6bf31394e508404336fd3562f3c: Uninitialized: clean time has not yet been initialized
I20260812 14:51:57.845394 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5417
I20260812 14:51:57.859012 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5656
I20260812 14:51:57.867465 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5888
I20260812 14:51:57.874554 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5210
2026-08-12T14:51:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2039 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[6180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[6180](info): set up 2 sockets
Aug 12 14:51:57 dist-test-slave-6zbq krb5kdc[6180](info): commencing operation
krb5kdc: starting...
W20260812 14:51:59.059804 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:51:59.166627 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:38087
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:38087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:59.244573 6201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:59.244752 6201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:59.244782 6201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:59.245869 6201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:51:59.245906 6201 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:51:59.245923 6201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:59.245940 6201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:59.245957 6201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:51:59.245977 6201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:51:59.247404 6201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:38087
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:38087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:59.247623 6201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:59.247809 6201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:59.252655 6213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:59.252621 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.252661 6201 server_base.cc:1034] running on GCE node
W20260812 14:51:59.252622 6214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.253077 6201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:59.253321 6201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:59.254452 6201 hybrid_clock.cc:648] HybridClock initialized: now 1786546319254440 us; error 29 us; skew 500 ppm
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:59.261848 6201 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:51:59.262249 6201 webserver.cc:460] Webserver started at http://127.4.27.62:46677/ using document root <none> and password file <none>
I20260812 14:51:59.262476 6201 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:59.262526 6201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:59.262647 6201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:59.263630 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "c66bd889feab4200a028e38d81afcbda"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "805278ac0a5bfa8a05c6fd3cd8cf86ac"
server_key_iv: "4b6f3f1637b3ee7c7ef6019e9a5ab97e"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.263947 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "c66bd889feab4200a028e38d81afcbda"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "805278ac0a5bfa8a05c6fd3cd8cf86ac"
server_key_iv: "4b6f3f1637b3ee7c7ef6019e9a5ab97e"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.265208 6201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:51:59.265865 6232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.266013 6201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:51:59.266098 6201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "c66bd889feab4200a028e38d81afcbda"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "805278ac0a5bfa8a05c6fd3cd8cf86ac"
server_key_iv: "4b6f3f1637b3ee7c7ef6019e9a5ab97e"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.266173 6201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:59.286921 6201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:59.288475 6201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:59.288601 6201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:59.293077 6201 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:38087
I20260812 14:51:59.293131 6326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:38087 every 8 connection(s)
I20260812 14:51:59.293411 6201 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:51:59.294020 6328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:59.296182 6328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: Bootstrap starting.
I20260812 14:51:59.296730 6328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:59.296984 6328 log.cc:824] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:59.297588 6328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: No bootstrap required, opened a new log
I20260812 14:51:59.299791 6328 raft_consensus.cc:348] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66bd889feab4200a028e38d81afcbda" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38087 } }
I20260812 14:51:59.299926 6328 raft_consensus.cc:374] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:59.299959 6328 raft_consensus.cc:729] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66bd889feab4200a028e38d81afcbda, State: Initialized, Role: FOLLOWER
I20260812 14:51:59.300094 6328 consensus_queue.cc:260] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66bd889feab4200a028e38d81afcbda" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38087 } }
I20260812 14:51:59.300170 6328 raft_consensus.cc:388] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:51:59.300222 6328 raft_consensus.cc:482] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:51:59.300273 6328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:51:59.300619 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6201
I20260812 14:51:59.300725 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:51:59.300889 4204 external_mini_cluster.cc:1428] Setting key aa7852862071d0a02fecd716f2e5ac86
I20260812 14:51:59.300904 6328 raft_consensus.cc:504] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66bd889feab4200a028e38d81afcbda" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38087 } }
I20260812 14:51:59.301007 6328 leader_election.cc:302] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c66bd889feab4200a028e38d81afcbda; no voters:
I20260812 14:51:59.301170 6328 leader_election.cc:288] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:51:59.301214 6332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:51:59.301589 6332 raft_consensus.cc:686] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [term 1 LEADER]: Becoming Leader. State: Replica: c66bd889feab4200a028e38d81afcbda, State: Running, Role: LEADER
I20260812 14:51:59.301651 6328 sys_catalog.cc:564] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:51:59.301760 6332 consensus_queue.cc:237] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66bd889feab4200a028e38d81afcbda" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38087 } }
I20260812 14:51:59.302047 6332 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [sys.catalog]: SysCatalogTable state changed. Reason: New leader c66bd889feab4200a028e38d81afcbda. Latest consensus state: current_term: 1 leader_uuid: "c66bd889feab4200a028e38d81afcbda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66bd889feab4200a028e38d81afcbda" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38087 } } }
I20260812 14:51:59.302206 6332 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [sys.catalog]: This master's current role is: LEADER
I20260812 14:51:59.302552 6343 sys_catalog.cc:455] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c66bd889feab4200a028e38d81afcbda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c66bd889feab4200a028e38d81afcbda" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38087 } } }
I20260812 14:51:59.302676 6343 sys_catalog.cc:458] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda [sys.catalog]: This master's current role is: LEADER
I20260812 14:51:59.302819 6356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:51:59.303386 6356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:51:59.305512 6356 catalog_manager.cc:1347] Generated new cluster ID: e92f56798beb48ef915b33a5c0201bba
I20260812 14:51:59.305563 6356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:59.311741 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.301336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.4.27.62:38087)
0812 14:51:59.301504 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.301513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.301565 (+ 52us) server_negotiation.cc:407] Connection header received
0812 14:51:59.301680 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.301686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.301739 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.301860 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.302643 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.303994 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.304902 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.305305 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.305398 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.307978 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.307982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.307982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.307993 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.310174 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.310594 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.310596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.310597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.310700 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.310899 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.310899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.310900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.311159 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.311297 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.311398 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.311524 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260812 14:51:59.313112 6356 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:51:59.313575 6356 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:51:59.320312 6356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: Generated new TSK 0
I20260812 14:51:59.320457 6356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:51:59.344910 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38087
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:59.423477 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:59.423664 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:59.423686 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:59.424763 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:59.424805 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:59.424824 6388 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:51:59.424840 6388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:51:59.426193 6388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:38087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:59.426393 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:59.426569 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:59.431169 6406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:59.431190 6402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.431219 6388 server_base.cc:1034] running on GCE node
W20260812 14:51:59.431190 6400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.431558 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:59.431766 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:59.432902 6388 hybrid_clock.cc:648] HybridClock initialized: now 1786546319432889 us; error 31 us; skew 500 ppm
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:59.435230 6388 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:51:59.435636 6388 webserver.cc:460] Webserver started at http://127.4.27.1:34049/ using document root <none> and password file <none>
I20260812 14:51:59.435804 6388 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:59.435842 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:59.435926 6388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:59.436878 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "3c0221b0db2a4c46bb0cf99aab6037fb"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "5769b897cde76b6f080a529e6b1c7f9b"
server_key_iv: "3a385b2bbece0652d62c71e8770a8b2a"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.437161 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "3c0221b0db2a4c46bb0cf99aab6037fb"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "5769b897cde76b6f080a529e6b1c7f9b"
server_key_iv: "3a385b2bbece0652d62c71e8770a8b2a"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.438261 6388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:51:59.438915 6419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.439064 6388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.439142 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "3c0221b0db2a4c46bb0cf99aab6037fb"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "5769b897cde76b6f080a529e6b1c7f9b"
server_key_iv: "3a385b2bbece0652d62c71e8770a8b2a"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.439194 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:59.449328 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:59.451259 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:59.451370 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:59.451774 6388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:59.451807 6388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.451828 6388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:59.451853 6388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.457773 6388 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37991
I20260812 14:51:59.457873 6624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37991 every 8 connection(s)
I20260812 14:51:59.458151 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:59.458885 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6388
I20260812 14:51:59.458983 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:51:59.459139 4204 external_mini_cluster.cc:1428] Setting key 7d4392bde7cd4145222078b4413655b1
I20260812 14:51:59.461565 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.452366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:51341 (local address 127.4.27.62:38087)
0812 14:51:59.452427 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.452429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.452594 (+ 165us) server_negotiation.cc:407] Connection header received
0812 14:51:59.453104 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.453105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.453120 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.453197 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.453952 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.454699 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.456147 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.456443 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.456518 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.458865 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.458867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.458867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.458878 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.460237 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.460604 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.460605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.460606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.460664 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.460884 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.460885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.460885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.460996 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.461058 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.461430 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.461502 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:51:59.461745 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.452334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.4.27.1:51341)
0812 14:51:59.452489 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.452526 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.452990 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.453281 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.453285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.453594 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:59.453853 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.453858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.454820 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.454821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.456050 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.456054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.456529 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.456530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.456628 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.457122 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:59.457132 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:59.458771 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:59.460322 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.460329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.460331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.460536 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.460716 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.460717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.460717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.460827 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.461102 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:59.461111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:59.461283 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.461391 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.461547 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260812 14:51:59.462210 6625 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38087
I20260812 14:51:59.462302 6625 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:59.462512 6625 heartbeater.cc:511] Master 127.4.27.62:38087 requested a full tablet report, sending...
I20260812 14:51:59.463056 6260 ts_manager.cc:194] Registered new tserver with Master: 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991)
I20260812 14:51:59.463759 6260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:51341
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
I20260812 14:51:59.468859 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.463372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:53347 (local address 127.4.27.62:38087)
0812 14:51:59.463413 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.463415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.463544 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:51:59.463649 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.463650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.463660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.463684 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.464180 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.464773 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.465723 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.465894 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.465937 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.466814 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.466815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.466815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.466821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.467983 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.468266 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.468266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.468267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.468312 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.468436 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.468437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.468437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.468526 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.468608 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.468748 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.468808 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:51:59.468861 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.463317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.4.27.1:53347)
0812 14:51:59.463476 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.463488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.463571 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.463738 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.463738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.464023 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:59.464092 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.464094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.464878 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.464879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.465652 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.465655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.465954 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.465955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.466026 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.466392 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:59.466397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:59.466734 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:59.468057 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.468058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.468059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.468209 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.468358 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.468359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.468359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.468402 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.468607 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:59.468607 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:59.468651 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.468699 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.468785 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:51:59.482672 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38087
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:59.560760 6650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:59.560941 6650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:59.560963 6650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:59.562130 6650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:59.562167 6650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:59.562187 6650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:51:59.562203 6650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:51:59.563611 6650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:38087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:59.563778 6650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:59.563941 6650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:59.568352 6660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:59.568409 6663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:59.568365 6659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.568588 6650 server_base.cc:1034] running on GCE node
I20260812 14:51:59.568789 6650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:59.568988 6650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:59.570144 6650 hybrid_clock.cc:648] HybridClock initialized: now 1786546319570133 us; error 27 us; skew 500 ppm
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:59.572455 6650 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:51:59.572855 6650 webserver.cc:460] Webserver started at http://127.4.27.2:42845/ using document root <none> and password file <none>
I20260812 14:51:59.573040 6650 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:59.573091 6650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:59.573213 6650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:59.574196 6650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "0c67f6a1898641a7a17ab7c1c044047e"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "ade996e064a42a106303ccc57c465ac9"
server_key_iv: "a796bc4e9034a623f41544bea802ceff"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.574548 6650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "0c67f6a1898641a7a17ab7c1c044047e"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "ade996e064a42a106303ccc57c465ac9"
server_key_iv: "a796bc4e9034a623f41544bea802ceff"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.575810 6650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:51:59.576485 6672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.576629 6650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:51:59.576712 6650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "0c67f6a1898641a7a17ab7c1c044047e"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "ade996e064a42a106303ccc57c465ac9"
server_key_iv: "a796bc4e9034a623f41544bea802ceff"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.576771 6650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:59.598308 6650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:59.599983 6650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:59.600095 6650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:59.600589 6650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:59.600631 6650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.600658 6650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:59.600675 6650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.607067 6650 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:41375
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:59.607192 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:41375 every 8 connection(s)
I20260812 14:51:59.607436 6650 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:51:59.609966 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.600989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:43529 (local address 127.4.27.62:38087)
0812 14:51:59.601056 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.601059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.601258 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:51:59.601789 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.601790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.601803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.601883 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.602787 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.603519 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.604952 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.605237 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.605314 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.607643 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.607645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.607645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.607661 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.608763 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.609112 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.609113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.609113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.609172 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.609382 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.609383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.609383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.609493 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.609550 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.609802 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.609882 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:51:59.610108 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.600944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.4.27.2:43529)
0812 14:51:59.601136 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.601174 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.601678 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.601953 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.601956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.602401 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:59.602690 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.602697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.603616 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.603617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.604860 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.604864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.605315 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.605315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.605407 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.605883 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:59.605893 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:59.607548 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:59.608836 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.608842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.608845 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.609046 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.609226 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.609226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.609227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.609327 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.609547 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:59.609553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:59.609683 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.609767 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.609909 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260812 14:51:59.610543 6880 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38087
I20260812 14:51:59.610623 6880 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:59.610786 6880 heartbeater.cc:511] Master 127.4.27.62:38087 requested a full tablet report, sending...
I20260812 14:51:59.611112 6261 ts_manager.cc:194] Registered new tserver with Master: 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375)
I20260812 14:51:59.611603 6261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:43529
I20260812 14:51:59.616190 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.611513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:47483 (local address 127.4.27.62:38087)
0812 14:51:59.611553 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.611555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.611602 (+ 47us) server_negotiation.cc:407] Connection header received
0812 14:51:59.611674 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.611674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.611683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.611734 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.612093 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.612622 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.613221 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.613385 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.613425 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.614073 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.614073 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.614073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.614079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.615250 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.615533 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.615534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.615535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.615602 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.615744 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.615744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.615745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.615868 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.615915 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.616076 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.616137 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:51:59.616209 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.611433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.4.27.2:47483)
0812 14:51:59.611548 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.611558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.611616 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.611775 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.611777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.611973 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:59.612019 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.612021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.612698 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.612699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.613167 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.613168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.613431 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.613431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.613480 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.613775 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:59.613778 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:59.614011 (+ 233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:59.615315 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.615316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.615316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.615456 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.615664 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.615665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.615665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.615708 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.615947 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:59.615948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:59.615990 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.616058 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.616139 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260812 14:51:59.616755 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6650
I20260812 14:51:59.616832 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:51:59.617013 4204 external_mini_cluster.cc:1428] Setting key 87c3bcca4e8e003a4929e6ef566c70e3
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:51:59.638376 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38087
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:51:59.717070 6895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:51:59.717245 6895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:51:59.717267 6895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:51:59.718334 6895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:51:59.718382 6895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:51:59.718401 6895 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:51:59.718420 6895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:51:59.719753 6895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:38087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:51:59.719913 6895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:51:59.720072 6895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:51:59.724851 6907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.724903 6895 server_base.cc:1034] running on GCE node
W20260812 14:51:59.724870 6909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:51:59.724870 6906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:51:59.725245 6895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:51:59.725440 6895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:51:59.726560 6895 hybrid_clock.cc:648] HybridClock initialized: now 1786546319726552 us; error 27 us; skew 500 ppm
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:51:59.728821 6895 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:51:59.729223 6895 webserver.cc:460] Webserver started at http://127.4.27.3:39259/ using document root <none> and password file <none>
I20260812 14:51:59.729408 6895 fs_manager.cc:359] Metadata directory not provided
I20260812 14:51:59.729453 6895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:51:59.729614 6895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:51:59.730649 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "982032903cbf4e30945033816c1e5e8a"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "246b4ad1ac4963c71a41f1d0d8e3cd65"
server_key_iv: "e30704fe99103309ba3f64cb81a14cc0"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.730976 6895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "982032903cbf4e30945033816c1e5e8a"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "246b4ad1ac4963c71a41f1d0d8e3cd65"
server_key_iv: "e30704fe99103309ba3f64cb81a14cc0"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.732301 6895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:51:59.733067 6918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.733355 6895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:51:59.733445 6895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "982032903cbf4e30945033816c1e5e8a"
format_stamp: "Formatted at 2026-08-12 14:51:59 on dist-test-slave-6zbq"
server_key: "246b4ad1ac4963c71a41f1d0d8e3cd65"
server_key_iv: "e30704fe99103309ba3f64cb81a14cc0"
server_key_version: "encryptionkey@0"
I20260812 14:51:59.733513 6895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:51:59.752053 6895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:51:59.754027 6895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:51:59.754158 6895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:51:59.754657 6895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:51:59.754698 6895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:51:59.754734 6895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:51:59.754760 6895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:59.761436 6895 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42929
I20260812 14:51:59.761519 7111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42929 every 8 connection(s)
I20260812 14:51:59.761827 6895 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:51:59.762473 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6895
I20260812 14:51:59.762574 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:51:59.762755 4204 external_mini_cluster.cc:1428] Setting key 0e4160fb866349ed306bdbfaf2c9e74f
I20260812 14:51:59.763831 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.755053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:43931 (local address 127.4.27.62:38087)
0812 14:51:59.755116 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.755119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.755282 (+ 163us) server_negotiation.cc:407] Connection header received
0812 14:51:59.755729 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.755731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.755743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.755820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.756405 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.756895 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.757842 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.758061 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.758117 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.760859 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.760860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.760861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.760869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.762417 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.762872 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.762874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.762874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.762938 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.763186 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.763187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.763188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.763344 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.763421 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.763702 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.763771 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:51:59.763949 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.755015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.4.27.3:43931)
0812 14:51:59.755167 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.755204 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.755646 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.755869 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.755873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.756126 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:59.756315 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.756319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.756975 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.756976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.757759 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.757761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.758129 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.758130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.758254 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.758749 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:59.758760 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:59.760775 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:59.762492 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.762497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.762499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.762779 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.763001 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.763002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.763002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.763128 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.763424 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:59.763430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:59.763578 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.763670 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.763784 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260812 14:51:59.764379 7112 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38087
I20260812 14:51:59.764459 7112 heartbeater.cc:464] Registering TS with master...
I20260812 14:51:59.764629 7112 heartbeater.cc:511] Master 127.4.27.62:38087 requested a full tablet report, sending...
I20260812 14:51:59.764950 6260 ts_manager.cc:194] Registered new tserver with Master: 982032903cbf4e30945033816c1e5e8a (127.4.27.3:42929)
I20260812 14:51:59.765319 6260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:43931
I20260812 14:51:59.769954 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.765377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:56757 (local address 127.4.27.62:38087)
0812 14:51:59.765415 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.765417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.765563 (+ 146us) server_negotiation.cc:407] Connection header received
0812 14:51:59.765612 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.765612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.765622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.765659 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.765994 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.766476 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.767107 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.767273 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.767313 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.767950 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.767950 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.767951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.767955 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.769088 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.769352 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.769353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.769354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.769396 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.769514 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.769515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.769515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.769613 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.769657 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.769850 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.769900 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:51:59.769986 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.765376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.4.27.3:56757)
0812 14:51:59.765496 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.765505 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.765560 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.765676 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.765676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.765875 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:51:59.765926 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.765927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.766565 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.766565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.767053 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.767054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.767313 (+ 259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.767314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.767359 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.767647 (+ 288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:51:59.767650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:51:59.767888 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:51:59.769151 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.769152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.769152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.769297 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.769439 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:51:59.769439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:51:59.769439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:51:59.769481 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:51:59.769692 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:51:59.769693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:51:59.769737 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.769835 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.769918 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260812 14:51:59.774185 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:51:59 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:51:59.794934 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.789314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.4.27.62:38087)
0812 14:51:59.789360 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.789362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.789371 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:51:59.789409 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.789410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.789419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.789476 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:51:59.789925 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.790400 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.791030 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.791196 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.791235 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.793043 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:51:59.793044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:51:59.793045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:51:59.793050 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:51:59.794168 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.794436 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.794437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.794437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.794483 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:51:59.794601 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:51:59.794601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:51:59.794601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:51:59.794691 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:51:59.794755 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.794825 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.794879 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:51:59.796229 6260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:51:59.797652 6260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:51:59.803929 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.799761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42929 (local address 127.0.0.1:46774)
0812 14:51:59.800107 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.800117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.800151 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.800427 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.800427 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.800429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:59.800457 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.800458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.801775 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.801776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.802958 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.802960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.803681 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.803682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.803737 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.803752 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.803797 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.803848 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":48,"threads_started":1}
I20260812 14:51:59.803977 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.800030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.4.27.1:37991)
0812 14:51:59.800207 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.800211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.800222 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:51:59.800261 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.800263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.800278 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.800363 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:59.800534 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.801780 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.802770 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.803565 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.803642 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.803696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.803847 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.803883 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260812 14:51:59.804091 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.799760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:41375 (local address 127.0.0.1:46934)
0812 14:51:59.799911 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.799948 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.800014 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.800473 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.800474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.800476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:59.800550 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.800552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.801622 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.801623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.802895 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.802898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.803878 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.803879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.803952 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.803956 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.803986 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.804023 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":6,"thread_start_us":54,"threads_started":1}
I20260812 14:51:59.803930 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.799758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37991 (local address 127.0.0.1:35754)
0812 14:51:59.799911 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:51:59.799949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:51:59.800016 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:51:59.800380 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:51:59.800383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:51:59.800385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:51:59.800449 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.800451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.801838 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.801839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.802688 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:51:59.802691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.803655 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.803656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:51:59.803737 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.803752 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 14:51:59.803820 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:51:59.803848 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260812 14:51:59.804379 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.800113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.4.27.3:42929)
0812 14:51:59.800274 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.800278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.800286 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:51:59.800322 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.800323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.800335 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.800522 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:59.800570 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.801682 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.803039 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.803584 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.804149 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.804218 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.804245 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.804283 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260812 14:51:59.804505 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:51:59.799953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.4.27.2:41375)
0812 14:51:59.800308 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0812 14:51:59.800312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:51:59.800320 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:51:59.800356 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:51:59.800357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:51:59.800371 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:51:59.800468 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:51:59.800611 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.801537 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.803121 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:51:59.803785 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:51:59.804279 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:51:59.804345 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:51:59.804371 (+ 26us) server_negotiation.cc:299] Negotiation successful
0812 14:51:59.804410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":72,"threads_started":1}
I20260812 14:51:59.804883 7012 tablet_service.cc:1467] Processing CreateTablet for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 (DEFAULT_TABLE table=test-table [id=b46fac211b344ffbb443208e87459501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:51:59.804890 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 (DEFAULT_TABLE table=test-table [id=b46fac211b344ffbb443208e87459501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:51:59.805035 6762 tablet_service.cc:1467] Processing CreateTablet for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 (DEFAULT_TABLE table=test-table [id=b46fac211b344ffbb443208e87459501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:51:59.805248 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:59.805248 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:59.805392 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:51:59.808439 7147 tablet_bootstrap.cc:492] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Bootstrap starting.
I20260812 14:51:59.808440 7144 tablet_bootstrap.cc:492] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Bootstrap starting.
I20260812 14:51:59.808771 7151 tablet_bootstrap.cc:492] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Bootstrap starting.
I20260812 14:51:59.809111 7144 tablet_bootstrap.cc:654] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:59.809265 7147 tablet_bootstrap.cc:654] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:59.809365 7151 tablet_bootstrap.cc:654] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Neither blocks nor log segments found. Creating new log.
I20260812 14:51:59.809474 7144 log.cc:824] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:59.809573 7147 log.cc:824] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:59.809684 7151 log.cc:824] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:51:59.810209 7144 tablet_bootstrap.cc:492] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: No bootstrap required, opened a new log
I20260812 14:51:59.810284 7144 ts_tablet_manager.cc:1397] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:51:59.810307 7147 tablet_bootstrap.cc:492] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: No bootstrap required, opened a new log
I20260812 14:51:59.810379 7147 ts_tablet_manager.cc:1397] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:51:59.810467 7151 tablet_bootstrap.cc:492] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: No bootstrap required, opened a new log
I20260812 14:51:59.810534 7151 ts_tablet_manager.cc:1397] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:51:59.812161 7151 raft_consensus.cc:348] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:51:59.812297 7151 raft_consensus.cc:374] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:59.812203 7144 raft_consensus.cc:348] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:51:59.812203 7147 raft_consensus.cc:348] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:51:59.812332 7151 raft_consensus.cc:729] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0221b0db2a4c46bb0cf99aab6037fb, State: Initialized, Role: FOLLOWER
I20260812 14:51:59.812359 7144 raft_consensus.cc:374] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:59.812359 7147 raft_consensus.cc:374] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:51:59.812393 7144 raft_consensus.cc:729] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c67f6a1898641a7a17ab7c1c044047e, State: Initialized, Role: FOLLOWER
I20260812 14:51:59.812393 7147 raft_consensus.cc:729] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982032903cbf4e30945033816c1e5e8a, State: Initialized, Role: FOLLOWER
I20260812 14:51:59.812458 7151 consensus_queue.cc:260] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:51:59.812500 7147 consensus_queue.cc:260] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:51:59.812500 7144 consensus_queue.cc:260] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:51:59.812690 7151 ts_tablet_manager.cc:1428] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:51:59.812754 7147 ts_tablet_manager.cc:1428] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:51:59.812754 7144 ts_tablet_manager.cc:1428] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:51:59.812877 7112 heartbeater.cc:503] Master 127.4.27.62:38087 was elected leader, sending a full tablet report...
I20260812 14:51:59.812893 6625 heartbeater.cc:503] Master 127.4.27.62:38087 was elected leader, sending a full tablet report...
I20260812 14:51:59.813129 6880 heartbeater.cc:503] Master 127.4.27.62:38087 was elected leader, sending a full tablet report...
W20260812 14:51:59.857975 6881 tablet.cc:2369] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:51:59.958851 6628 tablet.cc:2369] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:52:00.012413 7114 tablet.cc:2369] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:00.111925 7165 raft_consensus.cc:482] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:00.112042 7165 raft_consensus.cc:504] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.112360 7165 leader_election.cc:288] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375), 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991)
I20260812 14:52:00.116020 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.112558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:40555 (local address 127.4.27.2:41375)
0812 14:52:00.112602 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.112603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.112804 (+ 201us) server_negotiation.cc:407] Connection header received
0812 14:52:00.112831 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.112832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.112842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.112882 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:00.113016 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.113696 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.115134 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.115698 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.115764 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.115774 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.115918 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.115955 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:00.116020 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.112576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:59311 (local address 127.4.27.1:37991)
0812 14:52:00.112634 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.112637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.112647 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:00.112692 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.112693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.112705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.112756 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:00.112933 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.113678 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.115123 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.115690 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.115765 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.115774 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.115918 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.115953 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:52:00.116050 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.112535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37991 (local address 127.4.27.3:59311)
0812 14:52:00.112578 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:00.112590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:00.112635 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:00.112789 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:00.112790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:00.112791 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:00.112864 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.112865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.113761 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.113761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.115036 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.115040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.115783 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.115784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.115855 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.115858 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:00.115908 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:52:00.115951 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":68}
I20260812 14:52:00.116055 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.112535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:41375 (local address 127.4.27.3:40555)
0812 14:52:00.112730 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:00.112737 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:00.112783 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:00.112898 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:00.112899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:00.112900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:00.112961 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.112962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.113757 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.113758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.115037 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.115040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.115774 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.115775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.115846 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.115851 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:00.115908 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:52:00.115951 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":53,"thread_start_us":103,"threads_started":1}
I20260812 14:52:00.116441 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bb36fa4f9642b99a7a20c1c4d96ff7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" is_pre_election: true
I20260812 14:52:00.116539 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bb36fa4f9642b99a7a20c1c4d96ff7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67f6a1898641a7a17ab7c1c044047e" is_pre_election: true
I20260812 14:52:00.116601 6544 raft_consensus.cc:2454] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 0.
I20260812 14:52:00.116677 6798 raft_consensus.cc:2454] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 0.
I20260812 14:52:00.116834 6923 leader_election.cc:302] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0221b0db2a4c46bb0cf99aab6037fb, 982032903cbf4e30945033816c1e5e8a; no voters:
I20260812 14:52:00.116956 7165 raft_consensus.cc:2781] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:00.117015 7165 raft_consensus.cc:482] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:00.117031 7165 raft_consensus.cc:3037] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.118196 7165 raft_consensus.cc:504] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.118405 7165 leader_election.cc:288] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 election: Requested vote from peers 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375), 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991)
I20260812 14:52:00.118487 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bb36fa4f9642b99a7a20c1c4d96ff7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67f6a1898641a7a17ab7c1c044047e"
I20260812 14:52:00.118502 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58bb36fa4f9642b99a7a20c1c4d96ff7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb"
I20260812 14:52:00.118551 6798 raft_consensus.cc:3037] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.118577 6544 raft_consensus.cc:3037] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.119177 6798 raft_consensus.cc:2454] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 1.
I20260812 14:52:00.119259 6544 raft_consensus.cc:2454] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 1.
I20260812 14:52:00.119406 6924 leader_election.cc:302] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c67f6a1898641a7a17ab7c1c044047e, 982032903cbf4e30945033816c1e5e8a; no voters:
I20260812 14:52:00.119509 7165 raft_consensus.cc:2781] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:00.119637 7165 raft_consensus.cc:686] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 1 LEADER]: Becoming Leader. State: Replica: 982032903cbf4e30945033816c1e5e8a, State: Running, Role: LEADER
I20260812 14:52:00.119719 7165 consensus_queue.cc:237] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.120554 6261 catalog_manager.cc:5495] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 982032903cbf4e30945033816c1e5e8a (127.4.27.3). New cstate: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } health_report { overall_health: HEALTHY } } }
I20260812 14:52:00.155437 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.153154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46778 (local address 127.4.27.3:42929)
0812 14:52:00.153234 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.153236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.153246 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:00.153300 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.153301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.153320 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.153367 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:00.153494 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.153958 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.154720 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.154915 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.154975 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.155128 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:00.155227 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:00.155281 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.155318 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.155337 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 14:52:00.157166 6798 raft_consensus.cc:1261] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Refusing update from remote peer 982032903cbf4e30945033816c1e5e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:00.157166 6544 raft_consensus.cc:1261] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Refusing update from remote peer 982032903cbf4e30945033816c1e5e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:00.157493 7165 consensus_queue.cc:1035] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:00.157634 7169 consensus_queue.cc:1035] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:00.160209 7183 mvcc.cc:205] Tried to move back new op lower bound from 7317693727361204224 to 7317693727210852352. Current Snapshot: MvccSnapshot[applied={T|T < 7317693727361204224}]
I20260812 14:52:00.160804 7174 mvcc.cc:205] Tried to move back new op lower bound from 7317693727361204224 to 7317693727210852352. Current Snapshot: MvccSnapshot[applied={T|T < 7317693727361204224 or (T in {7317693727361204224})}]
I20260812 14:52:00.163403 6261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:00.163535 6261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:00.164384 6261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: Sending DeleteTablet for 3 replicas of tablet 58bb36fa4f9642b99a7a20c1c4d96ff7
I20260812 14:52:00.164777 7012 tablet_service.cc:1514] Processing DeleteTablet for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:00 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:46774
I20260812 14:52:00.164783 6762 tablet_service.cc:1514] Processing DeleteTablet for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:00 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:46934
I20260812 14:52:00.164968 6261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:00.165109 6261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:00.165143 7193 tablet_replica.cc:330] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: stopping tablet replica
I20260812 14:52:00.165141 6503 tablet_service.cc:1514] Processing DeleteTablet for tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:00 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:35754
I20260812 14:52:00.165253 7193 raft_consensus.cc:2229] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:00.165346 7193 raft_consensus.cc:2258] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:00.165535 7195 tablet_replica.cc:330] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: stopping tablet replica
I20260812 14:52:00.165627 7195 raft_consensus.cc:2229] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:00.165721 7195 raft_consensus.cc:2258] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:00.165843 7197 tablet_replica.cc:330] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: stopping tablet replica
I20260812 14:52:00.166200 7195 ts_tablet_manager.cc:1905] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:00.166452 7193 ts_tablet_manager.cc:1905] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:00.166932 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 0d763e882e344d2e9fb3e8fb3a225cf7 (DEFAULT_TABLE table=test-table [id=dd6eec0d552b418f81b53013352e8f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:00.167061 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d763e882e344d2e9fb3e8fb3a225cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:00.167963 7195 ts_tablet_manager.cc:1918] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:00.168068 7195 log.cc:1196] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/58bb36fa4f9642b99a7a20c1c4d96ff7
I20260812 14:52:00.168329 7195 ts_tablet_manager.cc:1939] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 982032903cbf4e30945033816c1e5e8a: Deleting consensus metadata
I20260812 14:52:00.168625 6762 tablet_service.cc:1467] Processing CreateTablet for tablet 0d763e882e344d2e9fb3e8fb3a225cf7 (DEFAULT_TABLE table=test-table [id=dd6eec0d552b418f81b53013352e8f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:00.168774 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d763e882e344d2e9fb3e8fb3a225cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:00.168790 7193 ts_tablet_manager.cc:1918] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:00.168787 6236 catalog_manager.cc:4841] TS 982032903cbf4e30945033816c1e5e8a (127.4.27.3:42929): tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 (table test-table [id=b46fac211b344ffbb443208e87459501]) successfully deleted
I20260812 14:52:00.168969 7193 log.cc:1196] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/58bb36fa4f9642b99a7a20c1c4d96ff7
I20260812 14:52:00.169420 7193 ts_tablet_manager.cc:1939] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 0c67f6a1898641a7a17ab7c1c044047e: Deleting consensus metadata
I20260812 14:52:00.169739 7144 tablet_bootstrap.cc:492] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Bootstrap starting.
I20260812 14:52:00.169806 7012 tablet_service.cc:1467] Processing CreateTablet for tablet 0d763e882e344d2e9fb3e8fb3a225cf7 (DEFAULT_TABLE table=test-table [id=dd6eec0d552b418f81b53013352e8f2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:00.169909 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d763e882e344d2e9fb3e8fb3a225cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:00.170128 7144 tablet_bootstrap.cc:654] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:00.170779 7144 tablet_bootstrap.cc:492] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: No bootstrap required, opened a new log
I20260812 14:52:00.170830 7144 ts_tablet_manager.cc:1397] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:00.170974 7147 tablet_bootstrap.cc:492] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Bootstrap starting.
I20260812 14:52:00.170987 7151 tablet_bootstrap.cc:492] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Bootstrap starting.
I20260812 14:52:00.170972 7144 raft_consensus.cc:348] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.171037 7144 raft_consensus.cc:374] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:00.171058 7144 raft_consensus.cc:729] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c67f6a1898641a7a17ab7c1c044047e, State: Initialized, Role: FOLLOWER
I20260812 14:52:00.171101 7144 consensus_queue.cc:260] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.171347 7151 tablet_bootstrap.cc:654] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:00.171363 7147 tablet_bootstrap.cc:654] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:00.171411 7144 ts_tablet_manager.cc:1428] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:52:00.171934 6238 catalog_manager.cc:4841] TS 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375): tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 (table test-table [id=b46fac211b344ffbb443208e87459501]) successfully deleted
I20260812 14:52:00.172087 7147 tablet_bootstrap.cc:492] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: No bootstrap required, opened a new log
I20260812 14:52:00.172132 7147 ts_tablet_manager.cc:1397] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:00.172282 7147 raft_consensus.cc:348] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.172355 7147 raft_consensus.cc:374] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:00.172375 7147 raft_consensus.cc:729] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982032903cbf4e30945033816c1e5e8a, State: Initialized, Role: FOLLOWER
I20260812 14:52:00.172402 7197 raft_consensus.cc:2229] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:00.172425 7147 consensus_queue.cc:260] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.172503 7147 ts_tablet_manager.cc:1428] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:00.172519 7197 raft_consensus.cc:2258] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:00.172966 7197 ts_tablet_manager.cc:1905] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:00.174367 7197 ts_tablet_manager.cc:1918] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:00.174396 7151 tablet_bootstrap.cc:492] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: No bootstrap required, opened a new log
I20260812 14:52:00.174433 7151 ts_tablet_manager.cc:1397] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260812 14:52:00.174443 7197 log.cc:1196] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/58bb36fa4f9642b99a7a20c1c4d96ff7
I20260812 14:52:00.174713 7197 ts_tablet_manager.cc:1939] T 58bb36fa4f9642b99a7a20c1c4d96ff7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Deleting consensus metadata
I20260812 14:52:00.174801 7151 raft_consensus.cc:348] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.174880 7151 raft_consensus.cc:374] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:00.174901 7151 raft_consensus.cc:729] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0221b0db2a4c46bb0cf99aab6037fb, State: Initialized, Role: FOLLOWER
I20260812 14:52:00.174947 7151 consensus_queue.cc:260] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.175062 7151 ts_tablet_manager.cc:1428] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:00.175107 6237 catalog_manager.cc:4841] TS 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991): tablet 58bb36fa4f9642b99a7a20c1c4d96ff7 (table test-table [id=b46fac211b344ffbb443208e87459501]) successfully deleted
I20260812 14:52:00.311730 7169 raft_consensus.cc:482] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:00.311834 7169 raft_consensus.cc:504] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.312008 7169 leader_election.cc:288] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991), 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375)
I20260812 14:52:00.312222 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d763e882e344d2e9fb3e8fb3a225cf7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" is_pre_election: true
I20260812 14:52:00.312227 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d763e882e344d2e9fb3e8fb3a225cf7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67f6a1898641a7a17ab7c1c044047e" is_pre_election: true
I20260812 14:52:00.312314 6544 raft_consensus.cc:2454] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 0.
I20260812 14:52:00.312323 6798 raft_consensus.cc:2454] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 0.
I20260812 14:52:00.312475 6923 leader_election.cc:302] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0221b0db2a4c46bb0cf99aab6037fb, 982032903cbf4e30945033816c1e5e8a; no voters:
I20260812 14:52:00.312575 7169 raft_consensus.cc:2781] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:00.312614 7169 raft_consensus.cc:482] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:00.312628 7169 raft_consensus.cc:3037] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.313324 7169 raft_consensus.cc:504] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.313445 7169 leader_election.cc:288] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 election: Requested vote from peers 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991), 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375)
I20260812 14:52:00.313611 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d763e882e344d2e9fb3e8fb3a225cf7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb"
I20260812 14:52:00.313623 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d763e882e344d2e9fb3e8fb3a225cf7" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67f6a1898641a7a17ab7c1c044047e"
I20260812 14:52:00.313690 6798 raft_consensus.cc:3037] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.313690 6544 raft_consensus.cc:3037] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.314463 6798 raft_consensus.cc:2454] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 1.
I20260812 14:52:00.314463 6544 raft_consensus.cc:2454] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 1.
I20260812 14:52:00.314617 6923 leader_election.cc:302] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0221b0db2a4c46bb0cf99aab6037fb, 982032903cbf4e30945033816c1e5e8a; no voters:
I20260812 14:52:00.314724 7169 raft_consensus.cc:2781] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:00.314792 7169 raft_consensus.cc:686] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 1 LEADER]: Becoming Leader. State: Replica: 982032903cbf4e30945033816c1e5e8a, State: Running, Role: LEADER
I20260812 14:52:00.314842 7169 consensus_queue.cc:237] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.315375 6261 catalog_manager.cc:5495] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 982032903cbf4e30945033816c1e5e8a (127.4.27.3). New cstate: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } health_report { overall_health: HEALTHY } } }
Aug 12 14:52:00 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:00.373396 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.366982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38087 (local address 127.0.0.1:41306)
0812 14:52:00.367126 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:00.367135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:00.367200 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:00.367329 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:00.367329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:00.367612 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:00.367662 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.367664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.368314 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.368315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.368851 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.368852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.369137 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.369138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.369183 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.369649 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:00.369654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:00.370950 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:00.372516 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:00.372523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:00.372525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:00.372731 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:00.372906 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:00.372906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:00.372907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:00.372959 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:00.373148 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:00.373153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:00.373195 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:52:00.373243 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:52:00.373326 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260812 14:52:00.373484 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.367063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.4.27.62:38087)
0812 14:52:00.367212 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.367213 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.367237 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:52:00.367257 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.367258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.367269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.367318 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:00.367718 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.368238 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.368907 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.369090 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.369135 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.371024 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:00.371025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:00.371025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:00.371031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:00.372414 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:00.372799 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:00.372799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:00.372800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:00.372857 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:00.373000 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:00.373000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:00.373000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:00.373089 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:00.373198 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.373273 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.373385 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260812 14:52:00.375473 6261 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:41306:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:52:00.375761 6261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:00.377223 7012 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7fb35ccaa44afaa4568dc67a10553 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3788fe409cc43be90a50805e3c1fd54]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:00.377213 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7fb35ccaa44afaa4568dc67a10553 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3788fe409cc43be90a50805e3c1fd54]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:00.377377 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7fb35ccaa44afaa4568dc67a10553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:00.377377 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7fb35ccaa44afaa4568dc67a10553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:00.378921 7147 tablet_bootstrap.cc:492] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: Bootstrap starting.
I20260812 14:52:00.379036 6762 tablet_service.cc:1467] Processing CreateTablet for tablet 30c7fb35ccaa44afaa4568dc67a10553 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e3788fe409cc43be90a50805e3c1fd54]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:00.379169 6762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c7fb35ccaa44afaa4568dc67a10553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:00.379300 7147 tablet_bootstrap.cc:654] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:00.379565 7151 tablet_bootstrap.cc:492] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: Bootstrap starting.
I20260812 14:52:00.379873 7147 tablet_bootstrap.cc:492] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: No bootstrap required, opened a new log
I20260812 14:52:00.379920 7147 ts_tablet_manager.cc:1397] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:00.380000 7151 tablet_bootstrap.cc:654] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:00.380031 7147 raft_consensus.cc:348] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.380097 7147 raft_consensus.cc:374] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:00.380116 7147 raft_consensus.cc:729] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982032903cbf4e30945033816c1e5e8a, State: Initialized, Role: FOLLOWER
I20260812 14:52:00.380158 7147 consensus_queue.cc:260] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.380278 7147 ts_tablet_manager.cc:1428] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:00.380300 7169 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.380389 7169 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.380424 7144 tablet_bootstrap.cc:492] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: Bootstrap starting.
I20260812 14:52:00.380849 7144 tablet_bootstrap.cc:654] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:00.381310 7151 tablet_bootstrap.cc:492] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: No bootstrap required, opened a new log
I20260812 14:52:00.381367 7151 ts_tablet_manager.cc:1397] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:00.381456 7144 tablet_bootstrap.cc:492] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: No bootstrap required, opened a new log
I20260812 14:52:00.381505 7144 ts_tablet_manager.cc:1397] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:00.381516 7151 raft_consensus.cc:348] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.381589 7151 raft_consensus.cc:374] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:00.381614 7151 raft_consensus.cc:729] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0221b0db2a4c46bb0cf99aab6037fb, State: Initialized, Role: FOLLOWER
I20260812 14:52:00.381633 7144 raft_consensus.cc:348] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.381656 7151 consensus_queue.cc:260] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.381696 7144 raft_consensus.cc:374] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:00.381723 7144 raft_consensus.cc:729] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c67f6a1898641a7a17ab7c1c044047e, State: Initialized, Role: FOLLOWER
I20260812 14:52:00.381742 7151 ts_tablet_manager.cc:1428] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:00.381768 7144 consensus_queue.cc:260] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.381860 7144 ts_tablet_manager.cc:1428] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:00.381886 7160 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.381961 7160 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.381974 7164 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.382043 7164 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.390534 7169 raft_consensus.cc:482] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:00.390587 7169 raft_consensus.cc:504] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.390712 7169 leader_election.cc:288] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991), 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375)
I20260812 14:52:00.390836 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7fb35ccaa44afaa4568dc67a10553" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" is_pre_election: true
I20260812 14:52:00.390849 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7fb35ccaa44afaa4568dc67a10553" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67f6a1898641a7a17ab7c1c044047e" is_pre_election: true
I20260812 14:52:00.390911 6798 raft_consensus.cc:2454] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 0.
I20260812 14:52:00.390908 6544 raft_consensus.cc:2454] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 0.
I20260812 14:52:00.391036 6924 leader_election.cc:302] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c67f6a1898641a7a17ab7c1c044047e, 982032903cbf4e30945033816c1e5e8a; no voters:
I20260812 14:52:00.391117 7169 raft_consensus.cc:2781] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:00.391167 7169 raft_consensus.cc:482] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:00.391187 7169 raft_consensus.cc:3037] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.391659 7169 raft_consensus.cc:504] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.391762 7169 leader_election.cc:288] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 election: Requested vote from peers 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991), 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375)
I20260812 14:52:00.391896 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7fb35ccaa44afaa4568dc67a10553" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb"
I20260812 14:52:00.391912 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c7fb35ccaa44afaa4568dc67a10553" candidate_uuid: "982032903cbf4e30945033816c1e5e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c67f6a1898641a7a17ab7c1c044047e"
I20260812 14:52:00.391957 6544 raft_consensus.cc:3037] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.391963 6798 raft_consensus.cc:3037] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:00.392441 6798 raft_consensus.cc:2454] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 1.
I20260812 14:52:00.392501 6544 raft_consensus.cc:2454] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982032903cbf4e30945033816c1e5e8a in term 1.
I20260812 14:52:00.392673 6923 leader_election.cc:302] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0221b0db2a4c46bb0cf99aab6037fb, 982032903cbf4e30945033816c1e5e8a; no voters:
I20260812 14:52:00.392827 7169 raft_consensus.cc:2781] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:00.392901 7169 raft_consensus.cc:686] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [term 1 LEADER]: Becoming Leader. State: Replica: 982032903cbf4e30945033816c1e5e8a, State: Running, Role: LEADER
I20260812 14:52:00.392947 7169 consensus_queue.cc:237] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } }
I20260812 14:52:00.393177 7165 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: TxnStatusTablet state changed. Reason: New leader 982032903cbf4e30945033816c1e5e8a. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.393283 7165 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:00.393428 6260 catalog_manager.cc:5495] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 982032903cbf4e30945033816c1e5e8a (127.4.27.3). New cstate: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } health_report { overall_health: HEALTHY } } }
I20260812 14:52:00.393755 7226 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:52:00.393880 7226 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:52:00.403211 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.400815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42929 (local address 127.0.0.1:46792)
0812 14:52:00.400888 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:00.400901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:00.400953 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:00.401244 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:00.401245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:00.401246 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:00.401288 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.401291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.401861 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.401862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.402537 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.402539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.402846 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.402847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.402900 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.402917 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:00.403058 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:00.403065 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:52:00.403109 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:52:00.403158 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:52:00.403239 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.401041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.4.27.3:42929)
0812 14:52:00.401118 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.401119 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.401126 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:00.401147 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.401148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.401156 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.401220 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:00.401348 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.401792 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.402609 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.402787 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.402841 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.402983 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:00.403012 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:00.403051 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.403119 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.403152 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:52:00.403995 6798 raft_consensus.cc:1261] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Refusing update from remote peer 982032903cbf4e30945033816c1e5e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:00.404037 6544 raft_consensus.cc:1261] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Refusing update from remote peer 982032903cbf4e30945033816c1e5e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:00.404201 7165 consensus_queue.cc:1035] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:00.404295 7169 consensus_queue.cc:1035] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:00.404775 7164 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: TxnStatusTablet state changed. Reason: New leader 982032903cbf4e30945033816c1e5e8a. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.404857 7164 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.404834 7160 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: TxnStatusTablet state changed. Reason: New leader 982032903cbf4e30945033816c1e5e8a. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.404919 7160 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.405637 7169 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.405720 7169 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:00.405767 7172 mvcc.cc:205] Tried to move back new op lower bound from 7317693728373530624 to 7317693728329854976. Current Snapshot: MvccSnapshot[applied={T|T < 7317693728373530624}]
I20260812 14:52:00.405980 7164 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.406056 7164 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 0c67f6a1898641a7a17ab7c1c044047e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.405649 7165 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.406437 7165 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 982032903cbf4e30945033816c1e5e8a: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:00.406785 7160 tablet_replica.cc:439] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "982032903cbf4e30945033816c1e5e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 } } peers { permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 } } peers { permanent_uuid: "982032903cbf4e30945033816c1e5e8a" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42929 } } }
I20260812 14:52:00.406868 7160 tablet_replica.cc:442] T 30c7fb35ccaa44afaa4568dc67a10553 P 3c0221b0db2a4c46bb0cf99aab6037fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:00.412056 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.409592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:42375 (local address 127.4.27.3:42929)
0812 14:52:00.409639 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.409640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.409787 (+ 147us) server_negotiation.cc:407] Connection header received
0812 14:52:00.409812 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.409813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.409821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.409868 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:00.409973 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.410445 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.411468 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.411640 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.411700 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.411876 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:00.411902 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:00.411935 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.411977 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.411995 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":29}
I20260812 14:52:00.412135 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.409531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42929 (local address 127.4.27.3:42375)
0812 14:52:00.409703 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:00.409712 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:00.409772 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:00.409871 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:00.409872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:00.409873 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:00.409925 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.409926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.410504 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.410505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.411397 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:00.411400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.411722 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.411723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:00.411786 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.411801 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:00.411937 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:00.411938 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:52:00.411978 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:52:00.412020 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260812 14:52:00.414270 6798 raft_consensus.cc:1261] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Refusing update from remote peer 982032903cbf4e30945033816c1e5e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:00.414357 6544 raft_consensus.cc:1261] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Refusing update from remote peer 982032903cbf4e30945033816c1e5e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:00.414546 7235 consensus_queue.cc:1035] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c67f6a1898641a7a17ab7c1c044047e" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 41375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:00.414635 7169 consensus_queue.cc:1035] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0221b0db2a4c46bb0cf99aab6037fb" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:00.427651 6260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:00.427739 6260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:00.428248 6260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c66bd889feab4200a028e38d81afcbda: Sending DeleteTablet for 3 replicas of tablet 0d763e882e344d2e9fb3e8fb3a225cf7
I20260812 14:52:00.428512 6503 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d763e882e344d2e9fb3e8fb3a225cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:00 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:35754
I20260812 14:52:00.428519 6762 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d763e882e344d2e9fb3e8fb3a225cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:00 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:46934
I20260812 14:52:00.428624 7197 tablet_replica.cc:330] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: stopping tablet replica
I20260812 14:52:00.428690 7197 raft_consensus.cc:2229] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:00.428728 7197 raft_consensus.cc:2258] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:00.428800 7193 tablet_replica.cc:330] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: stopping tablet replica
I20260812 14:52:00.428848 7193 raft_consensus.cc:2229] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:00.428880 7193 raft_consensus.cc:2258] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:00.428975 7197 ts_tablet_manager.cc:1905] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:00.429248 7012 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d763e882e344d2e9fb3e8fb3a225cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:00 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:46774
I20260812 14:52:00.429322 7195 tablet_replica.cc:330] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: stopping tablet replica
I20260812 14:52:00.429363 7195 raft_consensus.cc:2229] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:00.429406 7195 raft_consensus.cc:2258] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:00.429652 7195 ts_tablet_manager.cc:1905] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:00.430373 7197 ts_tablet_manager.cc:1918] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:00.430433 7197 log.cc:1196] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/0d763e882e344d2e9fb3e8fb3a225cf7
I20260812 14:52:00.430634 7197 ts_tablet_manager.cc:1939] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 3c0221b0db2a4c46bb0cf99aab6037fb: Deleting consensus metadata
I20260812 14:52:00.430905 6237 catalog_manager.cc:4841] TS 3c0221b0db2a4c46bb0cf99aab6037fb (127.4.27.1:37991): tablet 0d763e882e344d2e9fb3e8fb3a225cf7 (table test-table [id=dd6eec0d552b418f81b53013352e8f2d]) successfully deleted
I20260812 14:52:00.431159 7195 ts_tablet_manager.cc:1918] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:00.431227 7195 log.cc:1196] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/0d763e882e344d2e9fb3e8fb3a225cf7
I20260812 14:52:00.431429 7195 ts_tablet_manager.cc:1939] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 982032903cbf4e30945033816c1e5e8a: Deleting consensus metadata
I20260812 14:52:00.431700 6236 catalog_manager.cc:4841] TS 982032903cbf4e30945033816c1e5e8a (127.4.27.3:42929): tablet 0d763e882e344d2e9fb3e8fb3a225cf7 (table test-table [id=dd6eec0d552b418f81b53013352e8f2d]) successfully deleted
I20260812 14:52:00.431810 7193 ts_tablet_manager.cc:1905] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:00.432768 7193 ts_tablet_manager.cc:1918] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:00.432812 7193 log.cc:1196] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/0d763e882e344d2e9fb3e8fb3a225cf7
I20260812 14:52:00.432977 7193 ts_tablet_manager.cc:1939] T 0d763e882e344d2e9fb3e8fb3a225cf7 P 0c67f6a1898641a7a17ab7c1c044047e: Deleting consensus metadata
I20260812 14:52:00.433228 6238 catalog_manager.cc:4841] TS 0c67f6a1898641a7a17ab7c1c044047e (127.4.27.2:41375): tablet 0d763e882e344d2e9fb3e8fb3a225cf7 (table test-table [id=dd6eec0d552b418f81b53013352e8f2d]) successfully deleted
Aug 12 14:52:00 dist-test-slave-6zbq krb5kdc[6180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:52:00.443264 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.434574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.4.27.1:37991)
0812 14:52:00.434615 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.434617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.434747 (+ 130us) server_negotiation.cc:407] Connection header received
0812 14:52:00.434809 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.434810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.434818 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.434860 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:00.435515 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.436052 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.437121 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.437307 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.437352 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.440315 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:00.440324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:00.440324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:00.440330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:00.442077 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:00.442487 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:00.442491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:00.442491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:00.442571 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:00.442755 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:00.442756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:00.442756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:00.442909 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:00.443049 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.443085 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.443164 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:52:00.443504 6614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35770
I20260812 14:52:00.449702 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:00.444342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.4.27.62:38087)
0812 14:52:00.444388 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:00.444389 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:00.444398 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:00.444458 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:00.444459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:00.444468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:00.444511 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:00.444960 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.445425 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.446058 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:00.446266 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:00.446317 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:00.447558 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:00.447560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:00.447560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:00.447567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:00.448767 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:00.449044 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:00.449045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:00.449046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:00.449103 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:00.449266 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:00.449266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:00.449267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:00.449372 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:00.449453 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:00.449549 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:52:00.449621 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:52:00.449885 6260 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41310
I20260812 14:52:00.450762 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6388
I20260812 14:52:00.490527 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6650
I20260812 14:52:00.497929 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6895
I20260812 14:52:00.511608 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6201
2026-08-12T14:52:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2634 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:00 dist-test-slave-6zbq krb5kdc[7256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:00 dist-test-slave-6zbq krb5kdc[7256](info): set up 2 sockets
Aug 12 14:52:00 dist-test-slave-6zbq krb5kdc[7256](info): commencing operation
krb5kdc: starting...
W20260812 14:52:01.687114 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:01 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:01.793036 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46709
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:01.871830 7282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:01.872030 7282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:01.872059 7282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:01.873171 7282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:01.873210 7282 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:52:01.873229 7282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:01.873245 7282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:01.873270 7282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:01.873296 7282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:01.874758 7282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46709
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:01.874998 7282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:01.875192 7282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:01.879693 7297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:01.879755 7296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:01.879774 7282 server_base.cc:1034] running on GCE node
W20260812 14:52:01.879703 7303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:01.880167 7282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:01.880443 7282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:01.881577 7282 hybrid_clock.cc:648] HybridClock initialized: now 1786546321881566 us; error 31 us; skew 500 ppm
Aug 12 14:52:01 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:01.883868 7282 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:01.884279 7282 webserver.cc:460] Webserver started at http://127.4.27.62:37513/ using document root <none> and password file <none>
I20260812 14:52:01.884474 7282 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:01.884522 7282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:01.884644 7282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:01.885591 7282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "540cf9cd8b49427290e40498f42b99c4"
format_stamp: "Formatted at 2026-08-12 14:52:01 on dist-test-slave-6zbq"
server_key: "6cbecd238302df6ec94a0a5f64ccdc54"
server_key_iv: "6bf6a9f20a9861fc3dbaa5eafc6ef8e6"
server_key_version: "encryptionkey@0"
I20260812 14:52:01.885932 7282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "540cf9cd8b49427290e40498f42b99c4"
format_stamp: "Formatted at 2026-08-12 14:52:01 on dist-test-slave-6zbq"
server_key: "6cbecd238302df6ec94a0a5f64ccdc54"
server_key_iv: "6bf6a9f20a9861fc3dbaa5eafc6ef8e6"
server_key_version: "encryptionkey@0"
I20260812 14:52:01.887202 7282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:01.887840 7313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:01.887998 7282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:01.888080 7282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "540cf9cd8b49427290e40498f42b99c4"
format_stamp: "Formatted at 2026-08-12 14:52:01 on dist-test-slave-6zbq"
server_key: "6cbecd238302df6ec94a0a5f64ccdc54"
server_key_iv: "6bf6a9f20a9861fc3dbaa5eafc6ef8e6"
server_key_version: "encryptionkey@0"
I20260812 14:52:01.888139 7282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:01.915511 7282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:01.917125 7282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:01.917255 7282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:01.921576 7282 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46709
I20260812 14:52:01.921634 7419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46709 every 8 connection(s)
I20260812 14:52:01.921931 7282 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:01.922541 7420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:01.924717 7420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Bootstrap starting.
I20260812 14:52:01.925274 7420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:01.925529 7420 log.cc:824] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:01.926168 7420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: No bootstrap required, opened a new log
I20260812 14:52:01.927250 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7282
I20260812 14:52:01.927353 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:01.927512 4204 external_mini_cluster.cc:1428] Setting key 4694e709a928f544e36020754ee6f67e
I20260812 14:52:01.928587 7420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540cf9cd8b49427290e40498f42b99c4" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46709 } }
I20260812 14:52:01.928741 7420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:01.928781 7420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540cf9cd8b49427290e40498f42b99c4, State: Initialized, Role: FOLLOWER
I20260812 14:52:01.928903 7420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540cf9cd8b49427290e40498f42b99c4" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46709 } }
I20260812 14:52:01.928983 7420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:01.929019 7420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:01.929066 7420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:01.929642 7420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540cf9cd8b49427290e40498f42b99c4" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46709 } }
I20260812 14:52:01.929738 7420 leader_election.cc:302] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540cf9cd8b49427290e40498f42b99c4; no voters:
I20260812 14:52:01.929881 7420 leader_election.cc:288] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:01.929922 7429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:01.930058 7429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [term 1 LEADER]: Becoming Leader. State: Replica: 540cf9cd8b49427290e40498f42b99c4, State: Running, Role: LEADER
I20260812 14:52:01.930130 7420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:01.930243 7429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540cf9cd8b49427290e40498f42b99c4" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46709 } }
I20260812 14:52:01.930763 7430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540cf9cd8b49427290e40498f42b99c4. Latest consensus state: current_term: 1 leader_uuid: "540cf9cd8b49427290e40498f42b99c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540cf9cd8b49427290e40498f42b99c4" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46709 } } }
I20260812 14:52:01.930835 7430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:01.931155 7432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540cf9cd8b49427290e40498f42b99c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540cf9cd8b49427290e40498f42b99c4" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46709 } } }
I20260812 14:52:01.931206 7446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:01.931278 7432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:01.931766 7446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:01.933341 7446 catalog_manager.cc:1347] Generated new cluster ID: ee014713396b4f82b55d03a5bf86efe3
I20260812 14:52:01.933385 7446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:52:01 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:01.938472 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:01.928012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.4.27.62:46709)
0812 14:52:01.928191 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:01.928200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:01.928224 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:52:01.928561 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:01.928566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:01.928607 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:01.928709 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:01.929477 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:01.930703 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:01.931750 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:01.932343 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:01.932452 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:01.935070 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:01.935074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:01.935074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:01.935087 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:01.936918 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:01.937279 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:01.937282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:01.937282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:01.937416 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:01.937589 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:01.937590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:01.937590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:01.937881 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:01.938036 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:01.938100 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 14:52:01.938235 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260812 14:52:01.946504 7446 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:01.947149 7446 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:01.954003 7446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Generated new TSK 0
I20260812 14:52:01.954149 7446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:01.986476 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46709
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:02.064421 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:02.064602 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:02.064626 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:02.065726 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:02.065774 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:02.065797 7472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:52:02.065822 7472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:52:02.065850 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:02.067313 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:02.067534 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:02.067730 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:02.072275 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:02.072253 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:02.072346 7472 server_base.cc:1034] running on GCE node
W20260812 14:52:02.072254 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:02.072613 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:02.072831 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:02.073968 7472 hybrid_clock.cc:648] HybridClock initialized: now 1786546322073957 us; error 29 us; skew 500 ppm
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:02.076228 7472 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:02.076656 7472 webserver.cc:460] Webserver started at http://127.4.27.1:42271/ using document root <none> and password file <none>
I20260812 14:52:02.076853 7472 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:02.076900 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:02.077018 7472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:02.077950 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "2d2bc76b2f4d4d81b960c63561a10e92"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "4fa9ee1b87372934e437bf8d575fa191"
server_key_iv: "aa3ca82837328d5f231ff9c7f6250877"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.078258 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "2d2bc76b2f4d4d81b960c63561a10e92"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "4fa9ee1b87372934e437bf8d575fa191"
server_key_iv: "aa3ca82837328d5f231ff9c7f6250877"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.079439 7472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:02.080071 7501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.080216 7472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:02.080304 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "2d2bc76b2f4d4d81b960c63561a10e92"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "4fa9ee1b87372934e437bf8d575fa191"
server_key_iv: "aa3ca82837328d5f231ff9c7f6250877"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.080363 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:02.086987 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:02.088480 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:02.088600 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:02.089030 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:02.089074 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.089100 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:02.089118 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.095285 7472 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:46369
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:02.095338 7709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:46369 every 8 connection(s)
I20260812 14:52:02.095642 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:02.098145 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.089386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:49657 (local address 127.4.27.62:46709)
0812 14:52:02.089448 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.089451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.089614 (+ 163us) server_negotiation.cc:407] Connection header received
0812 14:52:02.090017 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.090019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.090034 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.090107 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.090897 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.091646 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.093038 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.093345 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.093416 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.095852 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.095853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.095854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.095864 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.097168 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.097484 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.097485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.097485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.097529 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.097696 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.097696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.097696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.097787 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.097830 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.098012 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.098081 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:52:02.098265 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.089355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.4.27.1:49657)
0812 14:52:02.089500 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.089536 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.089937 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.090160 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.090163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.090533 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.090794 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.090801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.091729 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.091730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.092944 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.092947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.093446 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.093447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.093572 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.094016 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.094025 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.095744 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.097233 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.097236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.097238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.097428 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.097572 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.097572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.097572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.097655 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.097828 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.097830 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.097927 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.097998 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.098101 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260812 14:52:02.098730 7711 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46709
I20260812 14:52:02.098809 7711 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:02.098963 7711 heartbeater.cc:511] Master 127.4.27.62:46709 requested a full tablet report, sending...
I20260812 14:52:02.099462 7342 ts_manager.cc:194] Registered new tserver with Master: 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369)
I20260812 14:52:02.100247 7342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:49657
I20260812 14:52:02.100716 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7472
I20260812 14:52:02.100816 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:02.100991 4204 external_mini_cluster.cc:1428] Setting key 6583c431ad1d031ece1d95a77d758bbb
I20260812 14:52:02.105031 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.099663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:45521 (local address 127.4.27.62:46709)
0812 14:52:02.099703 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.099705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.099822 (+ 117us) server_negotiation.cc:407] Connection header received
0812 14:52:02.099845 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.099846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.099855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.099915 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.100146 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.100754 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.101718 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.101944 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.101993 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.102882 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.102883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.102884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.102889 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.104075 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.104372 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.104373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.104374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.104425 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.104616 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.104617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.104618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.104719 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.104783 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.104918 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.104979 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:02.105077 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.099608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.4.27.1:45521)
0812 14:52:02.099756 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.099765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.099814 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.099919 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.099919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.100053 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.100097 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.100098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.100870 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.100871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.101599 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.101602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.102012 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.102013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.102079 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.102482 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.102487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.102812 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.104158 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.104159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.104159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.104310 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.104478 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.104479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.104479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.104537 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.104787 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.104788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.104831 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.104900 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.104988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:02.124064 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46709
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:02.201316 7730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:02.201489 7730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:02.201511 7730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:02.202626 7730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:02.202664 7730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:02.202685 7730 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:52:02.202708 7730 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:52:02.202737 7730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:02.204147 7730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:02.204358 7730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:02.204543 7730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:02.208933 7742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:02.208942 7743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:02.208999 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:02.209113 7730 server_base.cc:1034] running on GCE node
I20260812 14:52:02.209292 7730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:02.209514 7730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:02.210637 7730 hybrid_clock.cc:648] HybridClock initialized: now 1786546322210625 us; error 30 us; skew 500 ppm
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:02.212821 7730 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:02.213212 7730 webserver.cc:460] Webserver started at http://127.4.27.2:40917/ using document root <none> and password file <none>
I20260812 14:52:02.213408 7730 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:02.213454 7730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:02.213570 7730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:02.214524 7730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "605c453582ff4a0ebef177204ee18ac1"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "9308de0251ac1ffd2b5c081ef4f1931d"
server_key_iv: "cfadcf1f247669786338a1415da642c2"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.214838 7730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "605c453582ff4a0ebef177204ee18ac1"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "9308de0251ac1ffd2b5c081ef4f1931d"
server_key_iv: "cfadcf1f247669786338a1415da642c2"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.215982 7730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:02.216637 7754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.216779 7730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:02.216858 7730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "605c453582ff4a0ebef177204ee18ac1"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "9308de0251ac1ffd2b5c081ef4f1931d"
server_key_iv: "cfadcf1f247669786338a1415da642c2"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.216917 7730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:02.232391 7730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:02.233901 7730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:02.234021 7730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:02.234479 7730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:02.234520 7730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.234546 7730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:02.234575 7730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.240567 7730 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:39281
I20260812 14:52:02.240653 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:39281 every 8 connection(s)
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:02.240906 7730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:02.243342 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.234859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:42455 (local address 127.4.27.62:46709)
0812 14:52:02.234899 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.234901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.235090 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:52:02.235539 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.235540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.235553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.235603 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.236430 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.237174 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.238591 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.238896 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.238962 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.241270 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.241272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.241272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.241279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.242328 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.242695 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.242695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.242696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.242746 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.242911 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.242911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.242911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.242999 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.243044 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.243225 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.243283 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:02.243461 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.234855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.4.27.2:42455)
0812 14:52:02.234989 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.235020 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.235445 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.235665 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.235669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.235978 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.236323 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.236330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.237272 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.237273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.238485 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.238488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.238994 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.238994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.239139 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.239559 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.239568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.241167 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.242411 (+ 1244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.242417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.242419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.242622 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.242790 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.242790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.242790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.242872 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.243041 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.243044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.243139 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.243211 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.243315 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260812 14:52:02.243875 7949 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46709
I20260812 14:52:02.243953 7949 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:02.244107 7949 heartbeater.cc:511] Master 127.4.27.62:46709 requested a full tablet report, sending...
I20260812 14:52:02.244449 7345 ts_manager.cc:194] Registered new tserver with Master: 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281)
I20260812 14:52:02.244807 7345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:42455
I20260812 14:52:02.247968 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7730
I20260812 14:52:02.248040 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:02.248199 4204 external_mini_cluster.cc:1428] Setting key b922f4287b8635d701762234dedbb937
I20260812 14:52:02.249886 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.244741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.4.27.2:37153)
0812 14:52:02.244970 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.244981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.245035 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.245457 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.245458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.245597 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.245648 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.245650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.246205 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.246206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.246724 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.246726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.247001 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.247002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.247051 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.247348 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.247352 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.247586 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.249006 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.249006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.249007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.249157 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.249336 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.249336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.249337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.249379 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.249647 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.249647 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.249678 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.249721 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.249809 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260812 14:52:02.249909 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.244798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:37153 (local address 127.4.27.62:46709)
0812 14:52:02.244832 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.244834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.245033 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:52:02.245093 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.245093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.245101 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.245128 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.245695 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.246135 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.246781 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.246951 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.246997 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.247647 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.247648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.247648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.247653 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.248931 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.249220 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.249221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.249221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.249282 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.249430 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.249431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.249431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.249594 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.249658 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.249767 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.249833 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:02.269776 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46709
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:02.347810 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:02.347983 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:02.348006 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:02.349088 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:02.349128 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:02.349149 7964 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 14:52:02.349171 7964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:52:02.349200 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:02.350638 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:02.350834 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:02.351019 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:02.355491 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:02.355533 7975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:02.355537 7976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:02.355607 7964 server_base.cc:1034] running on GCE node
I20260812 14:52:02.355885 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:02.356063 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:02.357192 7964 hybrid_clock.cc:648] HybridClock initialized: now 1786546322357187 us; error 34 us; skew 500 ppm
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:02.359422 7964 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:02.359848 7964 webserver.cc:460] Webserver started at http://127.4.27.3:39781/ using document root <none> and password file <none>
I20260812 14:52:02.360033 7964 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:02.360080 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:02.360199 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:02.361128 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "7e67984214734ed18980603c71212641"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "ad1c520f3374d86fb341b1d79766e917"
server_key_iv: "95c541510dc6cb377e3c261531231e42"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.361441 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "7e67984214734ed18980603c71212641"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "ad1c520f3374d86fb341b1d79766e917"
server_key_iv: "95c541510dc6cb377e3c261531231e42"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.362640 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:02.363271 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.363411 7964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.363492 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "7e67984214734ed18980603c71212641"
format_stamp: "Formatted at 2026-08-12 14:52:02 on dist-test-slave-6zbq"
server_key: "ad1c520f3374d86fb341b1d79766e917"
server_key_iv: "95c541510dc6cb377e3c261531231e42"
server_key_version: "encryptionkey@0"
I20260812 14:52:02.363549 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:02.385268 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:02.386866 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:02.386991 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:02.387452 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:02.387491 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.387518 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:02.387544 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:02.393947 7964 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:38963
I20260812 14:52:02.394129 8198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:38963 every 8 connection(s)
I20260812 14:52:02.394317 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:02.396051 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.387801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:60993 (local address 127.4.27.62:46709)
0812 14:52:02.387852 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.387854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.388057 (+ 203us) server_negotiation.cc:407] Connection header received
0812 14:52:02.388465 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.388466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.388479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.388552 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.389115 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.389872 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.391372 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.391612 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.391676 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.393596 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.393597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.393597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.393605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.395052 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.395390 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.395390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.395391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.395436 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.395595 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.395595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.395595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.395694 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.395739 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.395928 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.395990 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:52:02.396173 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.387785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.4.27.3:60993)
0812 14:52:02.387938 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.387974 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.388376 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.388600 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.388602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.388813 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.389025 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.389030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.389974 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.389976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.391278 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.391281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.391699 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.391700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.391823 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.392257 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.392266 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.393513 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.395125 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.395128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.395129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.395330 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.395476 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.395476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.395476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.395556 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.395737 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.395739 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.395843 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.395914 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.396019 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260812 14:52:02.396596 8199 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46709
I20260812 14:52:02.396685 8199 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:02.396854 8199 heartbeater.cc:511] Master 127.4.27.62:46709 requested a full tablet report, sending...
I20260812 14:52:02.397202 7342 ts_manager.cc:194] Registered new tserver with Master: 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:02.397558 7342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:60993
I20260812 14:52:02.402171 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.397591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:44715 (local address 127.4.27.62:46709)
0812 14:52:02.397653 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.397655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.397701 (+ 46us) server_negotiation.cc:407] Connection header received
0812 14:52:02.397770 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.397771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.397780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.397824 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.398107 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.398615 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.399316 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.399502 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.399554 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.400294 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.400295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.400296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.400302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.401250 (+ 948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.401532 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.401533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.401534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.401583 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.401741 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.401742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.401742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.401840 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.401892 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.402024 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.402088 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:52:02.402194 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.397525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.4.27.3:44715)
0812 14:52:02.397642 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.397652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.397724 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.397827 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.397827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.397986 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.398041 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.398042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.398742 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.398743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.399244 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.399246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.399561 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.399562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.399635 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.399960 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.399964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.400213 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.401321 (+ 1108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.401322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.401322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.401473 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.401635 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.401636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.401636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.401692 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.401894 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.401895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.401937 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.401996 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.402089 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260812 14:52:02.403692 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7964
I20260812 14:52:02.403774 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:02.403928 4204 external_mini_cluster.cc:1428] Setting key 87367825195ef245996b9bfdbd4cc33d
I20260812 14:52:02.404636 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:02.425731 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.419863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.4.27.62:46709)
0812 14:52:02.419936 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.419938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.419948 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:02.419966 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.419967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.419975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.420024 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.420565 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.421003 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.421658 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.421821 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.421865 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.423890 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.423892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.423893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.423899 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.424974 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.425254 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.425255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.425255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.425296 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.425413 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.425413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.425413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.425489 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.425550 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.425623 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.425675 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 14:52:02.427134 7342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:02.428553 7342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:02.434604 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.430624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50530 (local address 127.4.27.3:38963)
0812 14:52:02.430799 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.430803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.430966 (+ 163us) server_negotiation.cc:407] Connection header received
0812 14:52:02.431024 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.431026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.431039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.431114 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:02.431441 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.432437 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.433517 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.434123 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.434187 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.434236 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.434486 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.434519 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260812 14:52:02.434594 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.430496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38963 (local address 127.0.0.1:50530)
0812 14:52:02.430898 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.430906 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.430960 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.431109 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.431110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.431112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:02.431165 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.431167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.432507 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.432508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.433447 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.433450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.434272 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.434273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.434375 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.434391 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.434455 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.434508 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":44,"thread_start_us":49,"threads_started":1}
I20260812 14:52:02.434933 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.430734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.4.27.1:46369)
0812 14:52:02.430920 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.430924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.430933 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:02.430966 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.430967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.430979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.431041 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:02.431191 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.432323 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.433765 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.434492 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.434547 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.434595 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.434806 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.434840 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
I20260812 14:52:02.435019 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.430711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.4.27.2:39281)
0812 14:52:02.431219 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.431223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.431232 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:02.431270 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.431272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.431287 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.431436 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:02.431607 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.432363 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.433817 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.434256 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.434318 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.434395 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.434636 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.434924 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":89,"threads_started":1}
I20260812 14:52:02.435281 8081 tablet_service.cc:1467] Processing CreateTablet for tablet 6edc612ef3d94aa2866f7d86d7adce40 (DEFAULT_TABLE table=test-table [id=930d8ed9bb7b4b19b7b96c7990854df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:02.435597 8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edc612ef3d94aa2866f7d86d7adce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:02.434895 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.430496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:46369 (local address 127.0.0.1:33652)
0812 14:52:02.430796 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.430805 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.430838 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.431066 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.431068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.431070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:02.431126 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.431128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.432404 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.432405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.433685 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.433688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.434629 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.434630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.434735 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.434739 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.434789 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.434833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":75,"thread_start_us":49,"threads_started":1}
I20260812 14:52:02.434598 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.430488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:39281 (local address 127.0.0.1:43670)
0812 14:52:02.430623 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.430655 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.430715 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.431489 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.431490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.431491 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:02.431529 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.431530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.432451 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.432451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.433730 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.433732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.434356 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.434357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.434418 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.434422 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.434469 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.434512 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":62,"thread_start_us":43,"threads_started":1}
I20260812 14:52:02.437961 8231 tablet_bootstrap.cc:492] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Bootstrap starting.
I20260812 14:52:02.438592 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 6edc612ef3d94aa2866f7d86d7adce40 (DEFAULT_TABLE table=test-table [id=930d8ed9bb7b4b19b7b96c7990854df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:02.438652 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 6edc612ef3d94aa2866f7d86d7adce40 (DEFAULT_TABLE table=test-table [id=930d8ed9bb7b4b19b7b96c7990854df7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:02.438902 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edc612ef3d94aa2866f7d86d7adce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:02.439018 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6edc612ef3d94aa2866f7d86d7adce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:02.439215 8231 tablet_bootstrap.cc:654] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:02.439543 8231 log.cc:824] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:02.440313 8231 tablet_bootstrap.cc:492] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: No bootstrap required, opened a new log
I20260812 14:52:02.440385 8231 ts_tablet_manager.cc:1397] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:02.441654 8236 tablet_bootstrap.cc:492] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Bootstrap starting.
I20260812 14:52:02.442195 8236 tablet_bootstrap.cc:654] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:02.442255 8231 raft_consensus.cc:348] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.442456 8231 raft_consensus.cc:374] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:02.442497 8231 raft_consensus.cc:729] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e67984214734ed18980603c71212641, State: Initialized, Role: FOLLOWER
I20260812 14:52:02.442601 8231 consensus_queue.cc:260] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.442694 8236 log.cc:824] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:02.442848 8231 ts_tablet_manager.cc:1428] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:52:02.442961 8199 heartbeater.cc:503] Master 127.4.27.62:46709 was elected leader, sending a full tablet report...
I20260812 14:52:02.443056 8238 tablet_bootstrap.cc:492] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Bootstrap starting.
I20260812 14:52:02.443375 8236 tablet_bootstrap.cc:492] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: No bootstrap required, opened a new log
I20260812 14:52:02.443432 8236 ts_tablet_manager.cc:1397] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:02.443642 8238 tablet_bootstrap.cc:654] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:02.443935 8238 log.cc:824] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:02.444550 8238 tablet_bootstrap.cc:492] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: No bootstrap required, opened a new log
I20260812 14:52:02.444600 8238 ts_tablet_manager.cc:1397] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:02.445108 8236 raft_consensus.cc:348] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.445226 8236 raft_consensus.cc:374] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:02.445256 8236 raft_consensus.cc:729] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605c453582ff4a0ebef177204ee18ac1, State: Initialized, Role: FOLLOWER
I20260812 14:52:02.445359 8236 consensus_queue.cc:260] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.445604 7949 heartbeater.cc:503] Master 127.4.27.62:46709 was elected leader, sending a full tablet report...
I20260812 14:52:02.445811 8236 ts_tablet_manager.cc:1428] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:02.446027 8238 raft_consensus.cc:348] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.446182 8238 raft_consensus.cc:374] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:02.446228 8238 raft_consensus.cc:729] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Initialized, Role: FOLLOWER
I20260812 14:52:02.446359 8238 consensus_queue.cc:260] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.446614 8238 ts_tablet_manager.cc:1428] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:02.446702 7711 heartbeater.cc:503] Master 127.4.27.62:46709 was elected leader, sending a full tablet report...
W20260812 14:52:02.491406 7951 tablet.cc:2369] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:52:02.596341 7712 tablet.cc:2369] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:52:02.644865 8202 tablet.cc:2369] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:02.652333 8239 raft_consensus.cc:482] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:02.652453 8239 raft_consensus.cc:504] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.652789 8239 leader_election.cc:288] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369)
I20260812 14:52:02.655511 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.653032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:60717 (local address 127.4.27.2:39281)
0812 14:52:02.653087 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.653089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.653098 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:02.653127 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.653128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.653139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.653194 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:02.653357 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.653866 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.654850 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.655202 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.655260 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.655271 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.655405 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.655442 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:52:02.655512 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.653028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:40425 (local address 127.4.27.1:46369)
0812 14:52:02.653073 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.653075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.653198 (+ 123us) server_negotiation.cc:407] Connection header received
0812 14:52:02.653242 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.653243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.653253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.653304 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:02.653441 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.653892 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.654862 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.655210 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.655271 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.655281 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.655410 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.655446 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:02.655529 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.652953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:39281 (local address 127.4.27.3:60717)
0812 14:52:02.653008 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.653019 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.653059 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.653217 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.653218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.653220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:02.653294 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.653296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.653936 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.653937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.654770 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.654772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.655274 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.655275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.655344 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.655347 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.655393 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.655435 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":48}
I20260812 14:52:02.655542 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.652957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:46369 (local address 127.4.27.3:40425)
0812 14:52:02.653120 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.653129 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.653195 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.653317 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.653317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.653319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:02.653386 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.653388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.653964 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.653964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.654773 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.654775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.655284 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.655285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.655345 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.655348 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.655394 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.655432 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":17,"thread_start_us":86,"threads_started":1}
I20260812 14:52:02.655876 7845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edc612ef3d94aa2866f7d86d7adce40" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1" is_pre_election: true
I20260812 14:52:02.655962 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edc612ef3d94aa2866f7d86d7adce40" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" is_pre_election: true
I20260812 14:52:02.656011 7845 raft_consensus.cc:2454] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 0.
I20260812 14:52:02.656111 7615 raft_consensus.cc:2454] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 0.
I20260812 14:52:02.656242 8001 leader_election.cc:302] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 605c453582ff4a0ebef177204ee18ac1, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:02.656345 8239 raft_consensus.cc:2781] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:02.656401 8239 raft_consensus.cc:482] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:02.656422 8239 raft_consensus.cc:3037] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:02.657467 8239 raft_consensus.cc:504] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.657584 8239 leader_election.cc:288] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 election: Requested vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369)
I20260812 14:52:02.657724 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edc612ef3d94aa2866f7d86d7adce40" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2bc76b2f4d4d81b960c63561a10e92"
I20260812 14:52:02.657801 7615 raft_consensus.cc:3037] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:02.658533 7615 raft_consensus.cc:2454] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 1.
I20260812 14:52:02.658630 7845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6edc612ef3d94aa2866f7d86d7adce40" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1"
I20260812 14:52:02.658689 8006 leader_election.cc:302] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:02.658715 7845 raft_consensus.cc:3037] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:02.658782 8239 raft_consensus.cc:2781] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:02.658941 8239 raft_consensus.cc:686] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 1 LEADER]: Becoming Leader. State: Replica: 7e67984214734ed18980603c71212641, State: Running, Role: LEADER
I20260812 14:52:02.659029 8239 consensus_queue.cc:237] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.659428 7845 raft_consensus.cc:2454] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 1.
I20260812 14:52:02.659898 7342 catalog_manager.cc:5495] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e67984214734ed18980603c71212641 (127.4.27.3). New cstate: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } health_report { overall_health: HEALTHY } } }
Aug 12 14:52:02 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:02.736871 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.730871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46709 (local address 127.0.0.1:39340)
0812 14:52:02.731028 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:02.731037 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:02.731089 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:02.731196 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:02.731197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:02.731504 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:02.731546 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.731548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.732167 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.732168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.732677 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:02.732679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.732946 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.732946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:02.732982 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.733448 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:02.733453 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:02.734702 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:02.735955 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.735961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.735963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.736154 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.736345 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:02.736345 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:02.736345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:02.736395 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:02.736626 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:02.736630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:02.736670 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:52:02.736719 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:52:02.736803 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260812 14:52:02.736960 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:02.730908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.4.27.62:46709)
0812 14:52:02.730956 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:02.730958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:02.731083 (+ 125us) server_negotiation.cc:407] Connection header received
0812 14:52:02.731126 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:02.731126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:02.731135 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:02.731187 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:02.731603 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.732100 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.732734 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:02.732903 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:02.732940 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:02.734771 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:02.734772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:02.734773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:02.734779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:02.735890 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.736232 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.736233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.736234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.736291 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:02.736472 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:02.736473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:02.736474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:02.736578 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:02.736639 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:02.736741 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 14:52:02.736860 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":28}
I20260812 14:52:02.738794 7342 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:39340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:52:02.738984 7342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:02.740260 7835 tablet_service.cc:1467] Processing CreateTablet for tablet e71037dafbf34555be5839ec4863d58d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d95218f7ed3437f962eb9ee86644031]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:02.740254 7589 tablet_service.cc:1467] Processing CreateTablet for tablet e71037dafbf34555be5839ec4863d58d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d95218f7ed3437f962eb9ee86644031]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:02.740275 8081 tablet_service.cc:1467] Processing CreateTablet for tablet e71037dafbf34555be5839ec4863d58d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d95218f7ed3437f962eb9ee86644031]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:02.740412 8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71037dafbf34555be5839ec4863d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:02.740412 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71037dafbf34555be5839ec4863d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:02.740412 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71037dafbf34555be5839ec4863d58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:02.741459 8236 tablet_bootstrap.cc:492] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: Bootstrap starting.
I20260812 14:52:02.741828 8231 tablet_bootstrap.cc:492] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: Bootstrap starting.
I20260812 14:52:02.741840 8238 tablet_bootstrap.cc:492] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: Bootstrap starting.
I20260812 14:52:02.742133 8236 tablet_bootstrap.cc:654] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:02.742156 8231 tablet_bootstrap.cc:654] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:02.742220 8238 tablet_bootstrap.cc:654] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:02.742810 8238 tablet_bootstrap.cc:492] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: No bootstrap required, opened a new log
I20260812 14:52:02.742852 8238 ts_tablet_manager.cc:1397] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:02.742971 8236 tablet_bootstrap.cc:492] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: No bootstrap required, opened a new log
I20260812 14:52:02.742978 8231 tablet_bootstrap.cc:492] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: No bootstrap required, opened a new log
I20260812 14:52:02.743010 8231 ts_tablet_manager.cc:1397] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:02.743007 8236 ts_tablet_manager.cc:1397] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:02.743005 8238 raft_consensus.cc:348] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.743069 8238 raft_consensus.cc:374] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:02.743089 8238 raft_consensus.cc:729] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Initialized, Role: FOLLOWER
I20260812 14:52:02.743139 8236 raft_consensus.cc:348] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.743119 8231 raft_consensus.cc:348] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.743161 8238 consensus_queue.cc:260] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.743196 8236 raft_consensus.cc:374] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:02.743201 8231 raft_consensus.cc:374] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:02.743214 8236 raft_consensus.cc:729] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605c453582ff4a0ebef177204ee18ac1, State: Initialized, Role: FOLLOWER
I20260812 14:52:02.743222 8231 raft_consensus.cc:729] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e67984214734ed18980603c71212641, State: Initialized, Role: FOLLOWER
I20260812 14:52:02.743269 8238 ts_tablet_manager.cc:1428] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.743259 8236 consensus_queue.cc:260] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.743260 8231 consensus_queue.cc:260] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:02.743351 8231 ts_tablet_manager.cc:1428] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.743340 8236 ts_tablet_manager.cc:1428] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:02.743422 8243 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:02.743489 8243 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:02.743474 8242 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:02.743562 8242 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:02.743520 8239 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:02.743615 8239 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:03.055055 8243 raft_consensus.cc:482] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:03.055169 8243 raft_consensus.cc:504] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.055531 8243 leader_election.cc:288] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:03.057287 8239 raft_consensus.cc:482] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:03.057358 8239 raft_consensus.cc:504] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.057487 8239 leader_election.cc:288] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369), 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281)
I20260812 14:52:03.057674 7845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71037dafbf34555be5839ec4863d58d" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1" is_pre_election: true
I20260812 14:52:03.057665 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71037dafbf34555be5839ec4863d58d" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" is_pre_election: true
I20260812 14:52:03.057747 7615 raft_consensus.cc:2454] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 0.
I20260812 14:52:03.057746 7845 raft_consensus.cc:2454] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 0.
I20260812 14:52:03.057881 8001 leader_election.cc:302] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 605c453582ff4a0ebef177204ee18ac1, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:03.058014 8239 raft_consensus.cc:2781] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:03.058043 8239 raft_consensus.cc:482] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:03.058059 8239 raft_consensus.cc:3037] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:03.058869 8239 raft_consensus.cc:504] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.059011 8239 leader_election.cc:288] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 election: Requested vote from peers 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369), 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281)
I20260812 14:52:03.059232 7845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71037dafbf34555be5839ec4863d58d" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1"
I20260812 14:52:03.059233 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71037dafbf34555be5839ec4863d58d" candidate_uuid: "7e67984214734ed18980603c71212641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2bc76b2f4d4d81b960c63561a10e92"
I20260812 14:52:03.059300 7845 raft_consensus.cc:3037] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:03.059300 7615 raft_consensus.cc:3037] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:03.059405 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.055702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:39281 (local address 127.4.27.1:57487)
0812 14:52:03.055834 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:03.055845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:03.055908 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:03.056042 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:03.056043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:03.056045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:03.056116 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.056119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.057048 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.057049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.058358 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.058362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.059082 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.059084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.059154 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.059159 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:03.059222 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:52:03.059273 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":49,"thread_start_us":71,"threads_started":1}
I20260812 14:52:03.059536 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.055702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38963 (local address 127.4.27.1:53029)
0812 14:52:03.055834 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:03.055845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:03.055908 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:03.056179 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:03.056180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:03.056182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:03.056228 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.056229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.057061 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.057062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.058613 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.058616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.059274 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.059275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.059335 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.059339 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:03.059381 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:52:03.059416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":46,"thread_start_us":72,"threads_started":1}
I20260812 14:52:03.059551 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.055754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:53029 (local address 127.4.27.3:38963)
0812 14:52:03.055863 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.055864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.055910 (+ 46us) server_negotiation.cc:407] Connection header received
0812 14:52:03.056077 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.056078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.056086 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.056147 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:03.056288 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.056993 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.058686 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.059174 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.059232 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.059243 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.059446 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.059492 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
I20260812 14:52:03.059787 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.055754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:57487 (local address 127.4.27.2:39281)
0812 14:52:03.055808 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.055810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.055896 (+ 86us) server_negotiation.cc:407] Connection header received
0812 14:52:03.055979 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.055980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.055988 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.056028 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:03.056197 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.056969 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.058439 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.058941 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.059014 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.059028 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.059701 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.059743 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:52:03.060022 7615 raft_consensus.cc:2454] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 1.
I20260812 14:52:03.060022 7845 raft_consensus.cc:2454] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e67984214734ed18980603c71212641 in term 1.
I20260812 14:52:03.060038 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71037dafbf34555be5839ec4863d58d" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1" is_pre_election: true
I20260812 14:52:03.060149 7870 raft_consensus.cc:2379] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d2bc76b2f4d4d81b960c63561a10e92 in current term 1: Already voted for candidate 7e67984214734ed18980603c71212641 in this term.
I20260812 14:52:03.060199 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71037dafbf34555be5839ec4863d58d" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641" is_pre_election: true
I20260812 14:52:03.060326 8120 raft_consensus.cc:2379] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d2bc76b2f4d4d81b960c63561a10e92 in current term 1: Already voted for candidate 7e67984214734ed18980603c71212641 in this term.
I20260812 14:52:03.060180 8001 leader_election.cc:302] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 605c453582ff4a0ebef177204ee18ac1, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:03.060516 8239 raft_consensus.cc:2781] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:03.060567 8239 raft_consensus.cc:686] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [term 1 LEADER]: Becoming Leader. State: Replica: 7e67984214734ed18980603c71212641, State: Running, Role: LEADER
I20260812 14:52:03.060567 7505 leader_election.cc:302] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92; no voters: 605c453582ff4a0ebef177204ee18ac1, 7e67984214734ed18980603c71212641
I20260812 14:52:03.060611 8239 consensus_queue.cc:237] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.060691 8243 raft_consensus.cc:2726] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 14:52:03.060870 8245 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: TxnStatusTablet state changed. Reason: New leader 7e67984214734ed18980603c71212641. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.060942 8245 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:03.061076 8269 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:52:03.061218 8269 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:52:03.061223 7345 catalog_manager.cc:5495] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e67984214734ed18980603c71212641 (127.4.27.3). New cstate: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } health_report { overall_health: HEALTHY } } }
I20260812 14:52:03.095458 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.092883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.4.27.3:38963)
0812 14:52:03.092940 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.092942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.092950 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:03.092981 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.092981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.093000 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.093065 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:03.093171 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.093886 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.094596 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.094900 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.094966 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.095096 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.095238 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.095311 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.095368 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.095389 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:52:03.095482 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.092695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38963 (local address 127.0.0.1:50532)
0812 14:52:03.092770 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:03.092780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:03.092818 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:03.093087 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:03.093088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:03.093089 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:03.093130 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.093133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.093948 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.093948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.094534 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.094536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.094961 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.094962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.095004 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.095018 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.095304 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.095315 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 14:52:03.095364 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:52:03.095412 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:52:03.097491 7615 raft_consensus.cc:1261] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Refusing update from remote peer 7e67984214734ed18980603c71212641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:03.097486 7870 raft_consensus.cc:1261] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Refusing update from remote peer 7e67984214734ed18980603c71212641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:03.097735 8245 consensus_queue.cc:1035] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:03.097847 8239 consensus_queue.cc:1035] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:03.098955 8243 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: TxnStatusTablet state changed. Reason: New leader 7e67984214734ed18980603c71212641. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.099036 8243 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:03.099159 8242 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: TxnStatusTablet state changed. Reason: New leader 7e67984214734ed18980603c71212641. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.099251 8242 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:03.100103 8239 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.100183 8239 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:03.100212 8243 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.100289 8243 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 2d2bc76b2f4d4d81b960c63561a10e92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:03.100601 8245 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.100675 8245 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 7e67984214734ed18980603c71212641: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:03.100911 8242 tablet_replica.cc:439] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e67984214734ed18980603c71212641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } } }
I20260812 14:52:03.100996 8242 tablet_replica.cc:442] T e71037dafbf34555be5839ec4863d58d P 605c453582ff4a0ebef177204ee18ac1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:03.101037 8277 mvcc.cc:205] Tried to move back new op lower bound from 7317693739404750848 to 7317693739256610816. Current Snapshot: MvccSnapshot[applied={T|T < 7317693739404750848}]
I20260812 14:52:03.101848 8273 mvcc.cc:205] Tried to move back new op lower bound from 7317693739404750848 to 7317693739256610816. Current Snapshot: MvccSnapshot[applied={T|T < 7317693739404750848 or (T in {7317693739404750848})}]
I20260812 14:52:03.104574 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.102488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.4.27.3:38963)
0812 14:52:03.102522 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.102524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.102633 (+ 109us) server_negotiation.cc:407] Connection header received
0812 14:52:03.102693 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.102694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.102702 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.102751 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:03.102863 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.103292 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.104006 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.104194 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.104245 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.104371 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.104401 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.104444 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.104501 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.104518 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:52:03.108961 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.106919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:39259 (local address 127.4.27.3:38963)
0812 14:52:03.106949 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.106950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.107062 (+ 112us) server_negotiation.cc:407] Connection header received
0812 14:52:03.107105 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.107106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.107114 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.107152 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:03.107282 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.107686 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.108384 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.108550 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.108599 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.108754 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.108784 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.108828 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.108883 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.108896 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260812 14:52:03.109027 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.106885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38963 (local address 127.4.27.3:39259)
0812 14:52:03.107004 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:03.107012 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:03.107064 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:03.107162 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:03.107162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:03.107174 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:03.107224 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.107226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.107748 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.107749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.108316 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.108318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.108605 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.108606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.108671 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.108684 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.108834 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.108835 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:52:03.108880 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:52:03.108917 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260812 14:52:03.110718 7615 raft_consensus.cc:1261] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Refusing update from remote peer 7e67984214734ed18980603c71212641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:03.110811 7870 raft_consensus.cc:1261] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Refusing update from remote peer 7e67984214734ed18980603c71212641: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:03.110991 8239 consensus_queue.cc:1035] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:03.111088 8245 consensus_queue.cc:1035] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:03.112486 8275 mvcc.cc:205] Tried to move back new op lower bound from 7317693739460431872 to 7317693737611952128. Current Snapshot: MvccSnapshot[applied={T|T < 7317693739460431872}]
I20260812 14:52:03.121959 7342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:03.122054 7342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:03.122795 7342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Sending DeleteTablet for 3 replicas of tablet 6edc612ef3d94aa2866f7d86d7adce40
I20260812 14:52:03.123165 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edc612ef3d94aa2866f7d86d7adce40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:03 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:33652
I20260812 14:52:03.123208 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edc612ef3d94aa2866f7d86d7adce40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:03 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260812 14:52:03.123386 8317 tablet_replica.cc:330] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: stopping tablet replica
I20260812 14:52:03.123497 8318 tablet_replica.cc:330] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: stopping tablet replica
I20260812 14:52:03.123504 8317 raft_consensus.cc:2229] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:03.123571 8317 raft_consensus.cc:2258] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:03.123567 7342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:03.123693 7342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:03.123682 8318 raft_consensus.cc:2229] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:03.123744 8318 raft_consensus.cc:2258] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:03.123950 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet 6edc612ef3d94aa2866f7d86d7adce40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:03 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:50530
I20260812 14:52:03.124192 8321 tablet_replica.cc:330] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: stopping tablet replica
I20260812 14:52:03.124321 8321 raft_consensus.cc:2229] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:03.124341 8318 ts_tablet_manager.cc:1905] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:03.124452 8321 raft_consensus.cc:2258] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:03.124841 8321 ts_tablet_manager.cc:1905] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:03.125150 7589 tablet_service.cc:1467] Processing CreateTablet for tablet f913b17ff2db4eec9f47864ec19bd2d1 (DEFAULT_TABLE table=test-table [id=7abbb24b925b46ad91309e9d346a3abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:03.125204 7835 tablet_service.cc:1467] Processing CreateTablet for tablet f913b17ff2db4eec9f47864ec19bd2d1 (DEFAULT_TABLE table=test-table [id=7abbb24b925b46ad91309e9d346a3abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:03.125255 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f913b17ff2db4eec9f47864ec19bd2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:03.125312 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f913b17ff2db4eec9f47864ec19bd2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:03.126484 8321 ts_tablet_manager.cc:1918] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:03.126504 8238 tablet_bootstrap.cc:492] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Bootstrap starting.
I20260812 14:52:03.126547 8321 log.cc:1196] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/6edc612ef3d94aa2866f7d86d7adce40
I20260812 14:52:03.126552 8236 tablet_bootstrap.cc:492] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Bootstrap starting.
I20260812 14:52:03.126808 8321 ts_tablet_manager.cc:1939] T 6edc612ef3d94aa2866f7d86d7adce40 P 7e67984214734ed18980603c71212641: Deleting consensus metadata
I20260812 14:52:03.126914 8236 tablet_bootstrap.cc:654] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:03.126914 8238 tablet_bootstrap.cc:654] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:03.127053 8079 tablet_service.cc:1467] Processing CreateTablet for tablet f913b17ff2db4eec9f47864ec19bd2d1 (DEFAULT_TABLE table=test-table [id=7abbb24b925b46ad91309e9d346a3abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:03.127143 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f913b17ff2db4eec9f47864ec19bd2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:03.127496 8236 tablet_bootstrap.cc:492] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: No bootstrap required, opened a new log
I20260812 14:52:03.127544 8236 ts_tablet_manager.cc:1397] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:03.127676 8236 raft_consensus.cc:348] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.127730 8236 raft_consensus.cc:374] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:03.127748 8236 raft_consensus.cc:729] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605c453582ff4a0ebef177204ee18ac1, State: Initialized, Role: FOLLOWER
I20260812 14:52:03.127784 8236 consensus_queue.cc:260] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.127864 8236 ts_tablet_manager.cc:1428] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:03.128091 8231 tablet_bootstrap.cc:492] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Bootstrap starting.
I20260812 14:52:03.128386 8231 tablet_bootstrap.cc:654] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:03.128677 7319 catalog_manager.cc:4841] TS 7e67984214734ed18980603c71212641 (127.4.27.3:38963): tablet 6edc612ef3d94aa2866f7d86d7adce40 (table test-table [id=930d8ed9bb7b4b19b7b96c7990854df7]) successfully deleted
I20260812 14:52:03.128945 8318 ts_tablet_manager.cc:1918] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:03.129070 8318 log.cc:1196] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/6edc612ef3d94aa2866f7d86d7adce40
I20260812 14:52:03.129230 8231 tablet_bootstrap.cc:492] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: No bootstrap required, opened a new log
I20260812 14:52:03.129282 8231 ts_tablet_manager.cc:1397] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:52:03.129359 8318 ts_tablet_manager.cc:1939] T 6edc612ef3d94aa2866f7d86d7adce40 P 605c453582ff4a0ebef177204ee18ac1: Deleting consensus metadata
I20260812 14:52:03.129416 8231 raft_consensus.cc:348] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.129456 8231 raft_consensus.cc:374] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:03.129468 8231 raft_consensus.cc:729] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e67984214734ed18980603c71212641, State: Initialized, Role: FOLLOWER
I20260812 14:52:03.129498 8231 consensus_queue.cc:260] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.129570 8231 ts_tablet_manager.cc:1428] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:03.129742 7316 catalog_manager.cc:4841] TS 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281): tablet 6edc612ef3d94aa2866f7d86d7adce40 (table test-table [id=930d8ed9bb7b4b19b7b96c7990854df7]) successfully deleted
I20260812 14:52:03.130028 8317 ts_tablet_manager.cc:1905] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:03.131255 8317 ts_tablet_manager.cc:1918] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:03.131325 8317 log.cc:1196] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/6edc612ef3d94aa2866f7d86d7adce40
I20260812 14:52:03.131480 8317 ts_tablet_manager.cc:1939] T 6edc612ef3d94aa2866f7d86d7adce40 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting consensus metadata
I20260812 14:52:03.131863 7321 catalog_manager.cc:4841] TS 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369): tablet 6edc612ef3d94aa2866f7d86d7adce40 (table test-table [id=930d8ed9bb7b4b19b7b96c7990854df7]) successfully deleted
I20260812 14:52:03.134703 8238 tablet_bootstrap.cc:492] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: No bootstrap required, opened a new log
I20260812 14:52:03.134745 8238 ts_tablet_manager.cc:1397] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260812 14:52:03.134855 8238 raft_consensus.cc:348] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.134903 8238 raft_consensus.cc:374] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:03.134915 8238 raft_consensus.cc:729] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Initialized, Role: FOLLOWER
I20260812 14:52:03.134945 8238 consensus_queue.cc:260] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.135025 8238 ts_tablet_manager.cc:1428] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:03.429883 8243 raft_consensus.cc:482] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:03.429977 8243 raft_consensus.cc:504] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.430152 8243 leader_election.cc:288] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:03.430392 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f913b17ff2db4eec9f47864ec19bd2d1" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641" is_pre_election: true
I20260812 14:52:03.430387 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f913b17ff2db4eec9f47864ec19bd2d1" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1" is_pre_election: true
I20260812 14:52:03.430492 7870 raft_consensus.cc:2454] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 0.
I20260812 14:52:03.430498 8120 raft_consensus.cc:2454] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 0.
I20260812 14:52:03.430665 7505 leader_election.cc:302] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:03.430787 8243 raft_consensus.cc:2781] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:03.430845 8243 raft_consensus.cc:482] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:03.430866 8243 raft_consensus.cc:3037] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:03.431600 8243 raft_consensus.cc:504] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.431725 8243 leader_election.cc:288] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 election: Requested vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:03.431869 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f913b17ff2db4eec9f47864ec19bd2d1" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641"
I20260812 14:52:03.431891 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f913b17ff2db4eec9f47864ec19bd2d1" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1"
I20260812 14:52:03.431938 8120 raft_consensus.cc:3037] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:03.431958 7870 raft_consensus.cc:3037] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:03.432565 7870 raft_consensus.cc:2454] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 1.
I20260812 14:52:03.432586 8120 raft_consensus.cc:2454] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 1.
I20260812 14:52:03.432745 7503 leader_election.cc:302] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 605c453582ff4a0ebef177204ee18ac1; no voters:
I20260812 14:52:03.432839 8243 raft_consensus.cc:2781] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:03.432958 8243 raft_consensus.cc:686] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Running, Role: LEADER
I20260812 14:52:03.433034 8243 consensus_queue.cc:237] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:03.433661 7342 catalog_manager.cc:5495] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:03.483526 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.481403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.4.27.1:46369)
0812 14:52:03.481514 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.481516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.481523 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:03.481556 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.481556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.481575 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.481644 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:03.481740 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.482198 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.482859 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.483094 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.483144 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.483236 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.483336 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.483399 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.483446 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.483461 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260812 14:52:03.487754 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.485741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:43101 (local address 127.4.27.3:38963)
0812 14:52:03.485795 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.485797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.485867 (+ 70us) server_negotiation.cc:407] Connection header received
0812 14:52:03.485913 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.485914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.485923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.485965 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:03.486052 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.486505 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.487157 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.487320 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.487363 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.487531 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.487565 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.487612 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.487661 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.487686 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:52:03.487957 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.485666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38963 (local address 127.4.27.1:43101)
0812 14:52:03.485820 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:03.485829 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:03.485877 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:03.485973 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:03.485973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:03.485975 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:03.486017 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.486018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.486552 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.486553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.487104 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.487105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.487387 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.487388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.487440 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.487456 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.487619 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.487621 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:03.487660 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 14:52:03.487699 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260812 14:52:03.489591 8120 raft_consensus.cc:1261] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Refusing update from remote peer 2d2bc76b2f4d4d81b960c63561a10e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:03.489591 7870 raft_consensus.cc:1261] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Refusing update from remote peer 2d2bc76b2f4d4d81b960c63561a10e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:03.489828 8328 consensus_queue.cc:1035] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:03.489945 8243 consensus_queue.cc:1035] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:03.496518 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.494580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:36617 (local address 127.4.27.1:46369)
0812 14:52:03.494612 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:03.494614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:03.494621 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:03.494659 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:03.494660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:03.494669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:03.494717 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:03.494813 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.495371 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.496006 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.496186 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.496224 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.496323 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.496343 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.496373 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:03.496445 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:52:03.496464 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:52:03.496546 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:03.494529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:46369 (local address 127.4.27.3:36617)
0812 14:52:03.494573 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:03.494581 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:03.494613 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:03.494723 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:03.494724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:03.494725 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:03.494759 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.494760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.495432 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.495433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.495946 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:03.495948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:03.496233 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:03.496234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:03.496272 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:03.496277 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:03.496391 (+ 114us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:03.496393 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:03.496429 (+ 36us) client_negotiation.cc:240] Negotiation successful
0812 14:52:03.496475 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:52:03.496877 8275 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 14:52:05.014771 7341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:05.014863 7341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:05.015616 7341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Sending DeleteTablet for 3 replicas of tablet f913b17ff2db4eec9f47864ec19bd2d1
I20260812 14:52:05.015844 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet f913b17ff2db4eec9f47864ec19bd2d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:05 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260812 14:52:05.015889 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet f913b17ff2db4eec9f47864ec19bd2d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:05 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:33652
I20260812 14:52:05.015930 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet f913b17ff2db4eec9f47864ec19bd2d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:05 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:50530
I20260812 14:52:05.016091 8379 tablet_replica.cc:330] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: stopping tablet replica
I20260812 14:52:05.016183 8379 raft_consensus.cc:2229] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:05.016249 8379 raft_consensus.cc:2258] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:05.017691 8379 ts_tablet_manager.cc:1905] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:05.019218 8380 tablet_replica.cc:330] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: stopping tablet replica
I20260812 14:52:05.019273 8380 raft_consensus.cc:2229] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:05.019338 8380 raft_consensus.cc:2258] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:05.019380 8379 ts_tablet_manager.cc:1918] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:52:05.019438 8379 log.cc:1196] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/f913b17ff2db4eec9f47864ec19bd2d1
I20260812 14:52:05.019443 8382 tablet_replica.cc:330] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: stopping tablet replica
I20260812 14:52:05.019491 8382 raft_consensus.cc:2229] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:05.019529 8382 raft_consensus.cc:2258] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:05.019693 8379 ts_tablet_manager.cc:1939] T f913b17ff2db4eec9f47864ec19bd2d1 P 605c453582ff4a0ebef177204ee18ac1: Deleting consensus metadata
I20260812 14:52:05.019766 8380 ts_tablet_manager.cc:1905] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:05.019804 8382 ts_tablet_manager.cc:1905] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:05.020117 7316 catalog_manager.cc:4841] TS 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281): tablet f913b17ff2db4eec9f47864ec19bd2d1 (table test-table [id=7abbb24b925b46ad91309e9d346a3abc]) successfully deleted
I20260812 14:52:05.021129 8380 ts_tablet_manager.cc:1918] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:52:05.021183 8380 log.cc:1196] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/f913b17ff2db4eec9f47864ec19bd2d1
I20260812 14:52:05.021365 8380 ts_tablet_manager.cc:1939] T f913b17ff2db4eec9f47864ec19bd2d1 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting consensus metadata
I20260812 14:52:05.021642 8382 ts_tablet_manager.cc:1918] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:52:05.021705 7321 catalog_manager.cc:4841] TS 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369): tablet f913b17ff2db4eec9f47864ec19bd2d1 (table test-table [id=7abbb24b925b46ad91309e9d346a3abc]) successfully deleted
I20260812 14:52:05.021734 8382 log.cc:1196] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/f913b17ff2db4eec9f47864ec19bd2d1
I20260812 14:52:05.021937 8382 ts_tablet_manager.cc:1939] T f913b17ff2db4eec9f47864ec19bd2d1 P 7e67984214734ed18980603c71212641: Deleting consensus metadata
I20260812 14:52:05.022205 7319 catalog_manager.cc:4841] TS 7e67984214734ed18980603c71212641 (127.4.27.3:38963): tablet f913b17ff2db4eec9f47864ec19bd2d1 (table test-table [id=7abbb24b925b46ad91309e9d346a3abc]) successfully deleted
Aug 12 14:52:05 dist-test-slave-6zbq krb5kdc[7256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:52:05 dist-test-slave-6zbq krb5kdc[7256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546325, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:05.043546 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:05.032135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.4.27.62:46709)
0812 14:52:05.032256 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:05.032258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:05.032267 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:05.032304 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:05.032305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:05.032314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:05.032366 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:05.032838 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:05.033326 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:05.033984 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:05.034162 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:05.034202 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:05.041459 (+ 7257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:05.041460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:05.041460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:05.041467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:05.042724 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:05.042987 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:05.042987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:05.042988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:05.043037 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:05.043194 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:05.043194 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:05.043195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:05.043289 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:05.043356 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:05.043426 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:52:05.043484 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260812 14:52:05.044803 7341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:05.044941 7341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:05.046072 7589 tablet_service.cc:1467] Processing CreateTablet for tablet b8283dac35e14ceb977f3a6a60ccad34 (DEFAULT_TABLE table=test-table [id=e06cc1f9a2484fb9876dd6ba10caa508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:05.046087 7835 tablet_service.cc:1467] Processing CreateTablet for tablet b8283dac35e14ceb977f3a6a60ccad34 (DEFAULT_TABLE table=test-table [id=e06cc1f9a2484fb9876dd6ba10caa508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:05.046192 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8283dac35e14ceb977f3a6a60ccad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:05.046192 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8283dac35e14ceb977f3a6a60ccad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:05.046423 8079 tablet_service.cc:1467] Processing CreateTablet for tablet b8283dac35e14ceb977f3a6a60ccad34 (DEFAULT_TABLE table=test-table [id=e06cc1f9a2484fb9876dd6ba10caa508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:05.046527 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8283dac35e14ceb977f3a6a60ccad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:05.047420 8406 tablet_bootstrap.cc:492] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Bootstrap starting.
I20260812 14:52:05.047609 8407 tablet_bootstrap.cc:492] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Bootstrap starting.
I20260812 14:52:05.047767 8406 tablet_bootstrap.cc:654] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:05.047955 8407 tablet_bootstrap.cc:654] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:05.048018 8401 tablet_bootstrap.cc:492] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Bootstrap starting.
I20260812 14:52:05.048381 8401 tablet_bootstrap.cc:654] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:05.048463 8407 tablet_bootstrap.cc:492] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: No bootstrap required, opened a new log
I20260812 14:52:05.048512 8407 ts_tablet_manager.cc:1397] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:05.048614 8407 raft_consensus.cc:348] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.048672 8407 raft_consensus.cc:374] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:05.048689 8407 raft_consensus.cc:729] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e67984214734ed18980603c71212641, State: Initialized, Role: FOLLOWER
I20260812 14:52:05.048709 8406 tablet_bootstrap.cc:492] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: No bootstrap required, opened a new log
I20260812 14:52:05.048766 8406 ts_tablet_manager.cc:1397] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:05.048756 8407 consensus_queue.cc:260] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.048861 8407 ts_tablet_manager.cc:1428] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:05.048870 8406 raft_consensus.cc:348] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.048938 8406 raft_consensus.cc:374] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:05.048964 8406 raft_consensus.cc:729] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Initialized, Role: FOLLOWER
I20260812 14:52:05.049022 8406 consensus_queue.cc:260] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.049117 8406 ts_tablet_manager.cc:1428] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:05.049307 8401 tablet_bootstrap.cc:492] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: No bootstrap required, opened a new log
I20260812 14:52:05.049355 8401 ts_tablet_manager.cc:1397] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:05.049489 8401 raft_consensus.cc:348] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.049556 8401 raft_consensus.cc:374] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:05.049598 8401 raft_consensus.cc:729] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605c453582ff4a0ebef177204ee18ac1, State: Initialized, Role: FOLLOWER
I20260812 14:52:05.049644 8401 consensus_queue.cc:260] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.049795 8401 ts_tablet_manager.cc:1428] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:05.388912 8364 raft_consensus.cc:482] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:05.389004 8364 raft_consensus.cc:504] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.389173 8364 leader_election.cc:288] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:05.389408 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8283dac35e14ceb977f3a6a60ccad34" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641" is_pre_election: true
I20260812 14:52:05.389408 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8283dac35e14ceb977f3a6a60ccad34" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1" is_pre_election: true
I20260812 14:52:05.389513 8120 raft_consensus.cc:2454] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 0.
I20260812 14:52:05.389519 7870 raft_consensus.cc:2454] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 0.
I20260812 14:52:05.389690 7503 leader_election.cc:302] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 605c453582ff4a0ebef177204ee18ac1; no voters:
I20260812 14:52:05.389810 8364 raft_consensus.cc:2781] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:05.389868 8364 raft_consensus.cc:482] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:05.389881 8364 raft_consensus.cc:3037] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:05.390668 8364 raft_consensus.cc:504] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.390786 8364 leader_election.cc:288] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 election: Requested vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:05.390929 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8283dac35e14ceb977f3a6a60ccad34" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641"
I20260812 14:52:05.390941 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8283dac35e14ceb977f3a6a60ccad34" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1"
I20260812 14:52:05.391000 8120 raft_consensus.cc:3037] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:05.391031 7870 raft_consensus.cc:3037] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:05.391589 8120 raft_consensus.cc:2454] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 1.
I20260812 14:52:05.391692 7870 raft_consensus.cc:2454] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 1.
I20260812 14:52:05.391754 7505 leader_election.cc:302] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:05.391837 8364 raft_consensus.cc:2781] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:05.391880 8364 raft_consensus.cc:686] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Running, Role: LEADER
I20260812 14:52:05.391927 8364 consensus_queue.cc:237] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.392498 7341 catalog_manager.cc:5495] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:05.400645 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:05.398612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.4.27.1:46369)
0812 14:52:05.398751 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:05.398752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:05.398759 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:05.398801 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:05.398802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:05.398813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:05.398887 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:05.398993 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:05.399455 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:05.400147 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:05.400336 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:05.400383 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:05.400479 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:05.400504 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:05.400531 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:05.400567 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:52:05.400582 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260812 14:52:05.402570 8120 raft_consensus.cc:1261] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Refusing update from remote peer 2d2bc76b2f4d4d81b960c63561a10e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:05.402570 7870 raft_consensus.cc:1261] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Refusing update from remote peer 2d2bc76b2f4d4d81b960c63561a10e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:05.402778 8364 consensus_queue.cc:1035] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:05.402868 8344 consensus_queue.cc:1035] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:05.412250 7341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:05.412322 7341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:05.412813 7341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Sending DeleteTablet for 3 replicas of tablet b8283dac35e14ceb977f3a6a60ccad34
I20260812 14:52:05.413007 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet b8283dac35e14ceb977f3a6a60ccad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:05 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:33652
I20260812 14:52:05.413041 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet b8283dac35e14ceb977f3a6a60ccad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:05 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:50530
I20260812 14:52:05.413064 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet b8283dac35e14ceb977f3a6a60ccad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:05 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260812 14:52:05.413094 8380 tablet_replica.cc:330] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: stopping tablet replica
I20260812 14:52:05.413149 8380 raft_consensus.cc:2229] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:05.413156 8379 tablet_replica.cc:330] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: stopping tablet replica
I20260812 14:52:05.413173 8382 tablet_replica.cc:330] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: stopping tablet replica
I20260812 14:52:05.413197 8380 raft_consensus.cc:2258] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:05.413204 8379 raft_consensus.cc:2229] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:05.413226 8382 raft_consensus.cc:2229] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:05.413239 8379 raft_consensus.cc:2258] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:05.413261 8382 raft_consensus.cc:2258] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:05.413496 7341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 14:52:05.413580 8382 ts_tablet_manager.cc:1905] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:05.413591 8379 ts_tablet_manager.cc:1905] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:05.413612 8380 ts_tablet_manager.cc:1905] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:52:05.413630 7341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:05.414811 8079 tablet_service.cc:1467] Processing CreateTablet for tablet 8e21ad996cf84560a4c606d98b51a5d3 (DEFAULT_TABLE table=test-table [id=cc252c7fba5a4857a0a11dfe5bd7e76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:05.414919 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e21ad996cf84560a4c606d98b51a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:05.415104 8380 ts_tablet_manager.cc:1918] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:05.415117 8382 ts_tablet_manager.cc:1918] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:05.415160 8380 log.cc:1196] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/b8283dac35e14ceb977f3a6a60ccad34
I20260812 14:52:05.415160 8382 log.cc:1196] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/b8283dac35e14ceb977f3a6a60ccad34
I20260812 14:52:05.415175 8379 ts_tablet_manager.cc:1918] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:05.415225 8379 log.cc:1196] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/b8283dac35e14ceb977f3a6a60ccad34
I20260812 14:52:05.415215 7589 tablet_service.cc:1467] Processing CreateTablet for tablet 8e21ad996cf84560a4c606d98b51a5d3 (DEFAULT_TABLE table=test-table [id=cc252c7fba5a4857a0a11dfe5bd7e76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:05.415310 7589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e21ad996cf84560a4c606d98b51a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:05.415367 8382 ts_tablet_manager.cc:1939] T b8283dac35e14ceb977f3a6a60ccad34 P 7e67984214734ed18980603c71212641: Deleting consensus metadata
I20260812 14:52:05.415372 8380 ts_tablet_manager.cc:1939] T b8283dac35e14ceb977f3a6a60ccad34 P 2d2bc76b2f4d4d81b960c63561a10e92: Deleting consensus metadata
I20260812 14:52:05.415438 8379 ts_tablet_manager.cc:1939] T b8283dac35e14ceb977f3a6a60ccad34 P 605c453582ff4a0ebef177204ee18ac1: Deleting consensus metadata
I20260812 14:52:05.416186 8406 tablet_bootstrap.cc:492] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92: Bootstrap starting.
I20260812 14:52:05.416256 7319 catalog_manager.cc:4841] TS 7e67984214734ed18980603c71212641 (127.4.27.3:38963): tablet b8283dac35e14ceb977f3a6a60ccad34 (table test-table [id=e06cc1f9a2484fb9876dd6ba10caa508]) successfully deleted
I20260812 14:52:05.416352 7321 catalog_manager.cc:4841] TS 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369): tablet b8283dac35e14ceb977f3a6a60ccad34 (table test-table [id=e06cc1f9a2484fb9876dd6ba10caa508]) successfully deleted
I20260812 14:52:05.416425 8406 tablet_bootstrap.cc:654] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:05.416837 8406 tablet_bootstrap.cc:492] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92: No bootstrap required, opened a new log
I20260812 14:52:05.416877 8406 ts_tablet_manager.cc:1397] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:52:05.416957 8406 raft_consensus.cc:348] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.417018 8406 raft_consensus.cc:374] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:05.417037 8406 raft_consensus.cc:729] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Initialized, Role: FOLLOWER
I20260812 14:52:05.417083 8406 consensus_queue.cc:260] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.417183 8406 ts_tablet_manager.cc:1428] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:05.417488 8407 tablet_bootstrap.cc:492] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Bootstrap starting.
I20260812 14:52:05.417613 7316 catalog_manager.cc:4841] TS 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281): tablet b8283dac35e14ceb977f3a6a60ccad34 (table test-table [id=e06cc1f9a2484fb9876dd6ba10caa508]) successfully deleted
I20260812 14:52:05.417719 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 8e21ad996cf84560a4c606d98b51a5d3 (DEFAULT_TABLE table=test-table [id=cc252c7fba5a4857a0a11dfe5bd7e76b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:05.417765 8407 tablet_bootstrap.cc:654] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:05.417816 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e21ad996cf84560a4c606d98b51a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:05.418332 8407 tablet_bootstrap.cc:492] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: No bootstrap required, opened a new log
I20260812 14:52:05.418407 8407 ts_tablet_manager.cc:1397] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:05.418507 8407 raft_consensus.cc:348] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.418567 8407 raft_consensus.cc:374] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:05.418591 8407 raft_consensus.cc:729] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e67984214734ed18980603c71212641, State: Initialized, Role: FOLLOWER
I20260812 14:52:05.418629 8407 consensus_queue.cc:260] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.418715 8407 ts_tablet_manager.cc:1428] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:05.418953 8401 tablet_bootstrap.cc:492] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Bootstrap starting.
I20260812 14:52:05.419376 8401 tablet_bootstrap.cc:654] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:05.419952 8401 tablet_bootstrap.cc:492] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: No bootstrap required, opened a new log
I20260812 14:52:05.420001 8401 ts_tablet_manager.cc:1397] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:05.420130 8401 raft_consensus.cc:348] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.420181 8401 raft_consensus.cc:374] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:05.420192 8401 raft_consensus.cc:729] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605c453582ff4a0ebef177204ee18ac1, State: Initialized, Role: FOLLOWER
I20260812 14:52:05.420226 8401 consensus_queue.cc:260] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.420317 8401 ts_tablet_manager.cc:1428] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:05.452347 8431 raft_consensus.cc:482] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:05.452425 8431 raft_consensus.cc:504] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.452582 8431 leader_election.cc:288] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:05.452754 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21ad996cf84560a4c606d98b51a5d3" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1" is_pre_election: true
I20260812 14:52:05.452749 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21ad996cf84560a4c606d98b51a5d3" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641" is_pre_election: true
I20260812 14:52:05.452831 8120 raft_consensus.cc:2454] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 0.
I20260812 14:52:05.452831 7870 raft_consensus.cc:2454] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 0.
I20260812 14:52:05.452977 7503 leader_election.cc:302] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 605c453582ff4a0ebef177204ee18ac1; no voters:
I20260812 14:52:05.453073 8431 raft_consensus.cc:2781] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:05.453109 8431 raft_consensus.cc:482] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:05.453128 8431 raft_consensus.cc:3037] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:05.453730 8431 raft_consensus.cc:504] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.453835 8431 leader_election.cc:288] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 election: Requested vote from peers 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281), 7e67984214734ed18980603c71212641 (127.4.27.3:38963)
I20260812 14:52:05.453946 7870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21ad996cf84560a4c606d98b51a5d3" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605c453582ff4a0ebef177204ee18ac1"
I20260812 14:52:05.454018 7870 raft_consensus.cc:3037] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:05.453989 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e21ad996cf84560a4c606d98b51a5d3" candidate_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e67984214734ed18980603c71212641"
I20260812 14:52:05.454052 8120 raft_consensus.cc:3037] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:05.454595 7870 raft_consensus.cc:2454] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 1.
I20260812 14:52:05.454617 8120 raft_consensus.cc:2454] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2bc76b2f4d4d81b960c63561a10e92 in term 1.
I20260812 14:52:05.454766 7505 leader_election.cc:302] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2bc76b2f4d4d81b960c63561a10e92, 7e67984214734ed18980603c71212641; no voters:
I20260812 14:52:05.454900 8431 raft_consensus.cc:2781] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:05.454957 8431 raft_consensus.cc:686] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2bc76b2f4d4d81b960c63561a10e92, State: Running, Role: LEADER
I20260812 14:52:05.455006 8431 consensus_queue.cc:237] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } }
I20260812 14:52:05.455493 7341 catalog_manager.cc:5495] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d2bc76b2f4d4d81b960c63561a10e92" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 46369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:05.466863 8120 raft_consensus.cc:1261] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Refusing update from remote peer 2d2bc76b2f4d4d81b960c63561a10e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:05.466863 7870 raft_consensus.cc:1261] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Refusing update from remote peer 2d2bc76b2f4d4d81b960c63561a10e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:05.467090 8344 consensus_queue.cc:1035] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "605c453582ff4a0ebef177204ee18ac1" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:05.467186 8431 consensus_queue.cc:1035] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e67984214734ed18980603c71212641" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:52:05.470714 8352 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 14:52:06.987074 7342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:06.987176 7342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:06.987922 7342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 540cf9cd8b49427290e40498f42b99c4: Sending DeleteTablet for 3 replicas of tablet 8e21ad996cf84560a4c606d98b51a5d3
I20260812 14:52:06.988174 7589 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e21ad996cf84560a4c606d98b51a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:06 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:33652
I20260812 14:52:06.988185 8079 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e21ad996cf84560a4c606d98b51a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:06 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:50530
I20260812 14:52:06.988374 8480 tablet_replica.cc:330] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92: stopping tablet replica
I20260812 14:52:06.988435 8480 raft_consensus.cc:2229] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:06.988472 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7472
I20260812 14:52:06.988520 8480 raft_consensus.cc:2258] T 8e21ad996cf84560a4c606d98b51a5d3 P 2d2bc76b2f4d4d81b960c63561a10e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:06.988924 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e21ad996cf84560a4c606d98b51a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:06 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260812 14:52:06.994586 8481 tablet_replica.cc:330] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: stopping tablet replica
I20260812 14:52:06.994649 8481 raft_consensus.cc:2229] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:06.994679 8481 raft_consensus.cc:2258] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:06.995862 8481 ts_tablet_manager.cc:1905] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:06.997364 8481 ts_tablet_manager.cc:1918] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:52:06.997416 8481 log.cc:1196] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/8e21ad996cf84560a4c606d98b51a5d3
I20260812 14:52:06.997565 8481 ts_tablet_manager.cc:1939] T 8e21ad996cf84560a4c606d98b51a5d3 P 7e67984214734ed18980603c71212641: Deleting consensus metadata
I20260812 14:52:06.997882 7319 catalog_manager.cc:4841] TS 7e67984214734ed18980603c71212641 (127.4.27.3:38963): tablet 8e21ad996cf84560a4c606d98b51a5d3 (table test-table [id=cc252c7fba5a4857a0a11dfe5bd7e76b]) successfully deleted
I20260812 14:52:07.002599 8482 tablet_replica.cc:330] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: stopping tablet replica
I20260812 14:52:07.002691 8482 raft_consensus.cc:2229] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:07.002750 8482 raft_consensus.cc:2258] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:07.003003 8482 ts_tablet_manager.cc:1905] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:07.004480 8482 ts_tablet_manager.cc:1918] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 14:52:07.004549 8482 log.cc:1196] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/8e21ad996cf84560a4c606d98b51a5d3
I20260812 14:52:07.004760 8482 ts_tablet_manager.cc:1939] T 8e21ad996cf84560a4c606d98b51a5d3 P 605c453582ff4a0ebef177204ee18ac1: Deleting consensus metadata
I20260812 14:52:07.005059 7316 catalog_manager.cc:4841] TS 605c453582ff4a0ebef177204ee18ac1 (127.4.27.2:39281): tablet 8e21ad996cf84560a4c606d98b51a5d3 (table test-table [id=cc252c7fba5a4857a0a11dfe5bd7e76b]) successfully deleted
W20260812 14:52:07.006399 7321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:46369 (error 108)
I20260812 14:52:07.006542 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7730
I20260812 14:52:07.011324 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:07.011104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:46369 (local address 127.0.0.1:39984)
0812 14:52:07.011224 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:07.011278 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
W20260812 14:52:07.011474 7321 catalog_manager.cc:4584] TS 2d2bc76b2f4d4d81b960c63561a10e92 (127.4.27.1:46369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e21ad996cf84560a4c606d98b51a5d3: Network error: Client connection negotiation failed: client connection to 127.4.27.1:46369: connect: Connection refused (error 111)
I20260812 14:52:07.017761 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7964
I20260812 14:52:07.026854 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7282
2026-08-12T14:52:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6516 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:07 dist-test-slave-6zbq krb5kdc[8489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:07 dist-test-slave-6zbq krb5kdc[8489](info): set up 2 sockets
Aug 12 14:52:07 dist-test-slave-6zbq krb5kdc[8489](info): commencing operation
krb5kdc: starting...
W20260812 14:52:08.206317 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:08.312088 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42273
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:08.391110 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:08.391299 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:08.391330 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:08.392436 8520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:08.392477 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:08.392496 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:08.392511 8520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:08.392529 8520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:08.394009 8520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:42273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:08.394248 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:08.394479 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:08.398916 8528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.398940 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.398916 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:08.398983 8520 server_base.cc:1034] running on GCE node
I20260812 14:52:08.399333 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:08.399590 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:08.400729 8520 hybrid_clock.cc:648] HybridClock initialized: now 1786546328400715 us; error 30 us; skew 500 ppm
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:08.403054 8520 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:08.403573 8520 webserver.cc:460] Webserver started at http://127.4.27.62:43035/ using document root <none> and password file <none>
I20260812 14:52:08.403790 8520 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:08.403839 8520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:08.403961 8520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:08.404935 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "f4661d4b17384048894c7a7bd33a2a65"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "e5434701a071e1e4c8fa6517ee8c8f39"
server_key_iv: "2152fca81d484d234c28d4a1c84fa180"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.405246 8520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "f4661d4b17384048894c7a7bd33a2a65"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "e5434701a071e1e4c8fa6517ee8c8f39"
server_key_iv: "2152fca81d484d234c28d4a1c84fa180"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.406481 8520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:08.407120 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.407275 8520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.407356 8520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "f4661d4b17384048894c7a7bd33a2a65"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "e5434701a071e1e4c8fa6517ee8c8f39"
server_key_iv: "2152fca81d484d234c28d4a1c84fa180"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.407419 8520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:08.425602 8520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:08.427287 8520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:08.427412 8520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:08.431155 8520 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:42273
I20260812 14:52:08.431192 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:42273 every 8 connection(s)
I20260812 14:52:08.431479 8520 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:08.432258 8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:08.434377 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65: Bootstrap starting.
I20260812 14:52:08.434876 8632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:08.435122 8632 log.cc:824] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:08.435688 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65: No bootstrap required, opened a new log
I20260812 14:52:08.435956 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8520
I20260812 14:52:08.436049 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:08.436225 4204 external_mini_cluster.cc:1428] Setting key cf696d2b8a5bcbcee2d04f3dc4a6a513
I20260812 14:52:08.437626 8632 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4661d4b17384048894c7a7bd33a2a65" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42273 } }
I20260812 14:52:08.437770 8632 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:08.437808 8632 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4661d4b17384048894c7a7bd33a2a65, State: Initialized, Role: FOLLOWER
I20260812 14:52:08.437933 8632 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4661d4b17384048894c7a7bd33a2a65" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42273 } }
I20260812 14:52:08.438014 8632 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:08.438052 8632 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:08.438109 8632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:08.439062 8632 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4661d4b17384048894c7a7bd33a2a65" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42273 } }
I20260812 14:52:08.439162 8632 leader_election.cc:302] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4661d4b17384048894c7a7bd33a2a65; no voters:
I20260812 14:52:08.439308 8632 leader_election.cc:288] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:08.439352 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:08.439504 8639 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [term 1 LEADER]: Becoming Leader. State: Replica: f4661d4b17384048894c7a7bd33a2a65, State: Running, Role: LEADER
I20260812 14:52:08.439566 8632 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:08.440204 8639 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4661d4b17384048894c7a7bd33a2a65" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42273 } }
I20260812 14:52:08.440917 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4661d4b17384048894c7a7bd33a2a65. Latest consensus state: current_term: 1 leader_uuid: "f4661d4b17384048894c7a7bd33a2a65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4661d4b17384048894c7a7bd33a2a65" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42273 } } }
I20260812 14:52:08.441022 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:08.441356 8668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:08.441377 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4661d4b17384048894c7a7bd33a2a65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4661d4b17384048894c7a7bd33a2a65" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42273 } } }
I20260812 14:52:08.441644 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:08.442077 8668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:08.443794 8668 catalog_manager.cc:1347] Generated new cluster ID: fb65e75e37744433974620386fc4d7fc
I20260812 14:52:08.443842 8668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:08.447424 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.436677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.4.27.62:42273)
0812 14:52:08.436847 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:08.436855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:08.436878 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:52:08.437596 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:08.437601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:08.437642 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:08.437746 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:08.438541 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.439568 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.440477 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.441004 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.441149 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.444054 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:08.444058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:08.444059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:08.444069 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:08.445839 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.446223 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.446225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.446226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.446378 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.446573 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.446573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.446574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.446857 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:08.447050 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:08.447114 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 14:52:08.447242 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260812 14:52:08.454706 8668 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:08.455129 8668 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:08.458185 8668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4661d4b17384048894c7a7bd33a2a65: Generated new TSK 0
I20260812 14:52:08.458333 8668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:08.490312 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42273
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:08.568915 8676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:08.569097 8676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:08.569118 8676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:08.570190 8676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:08.570221 8676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:08.570246 8676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:08.571595 8676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:08.571769 8676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:08.571935 8676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:08.576488 8681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.576473 8686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.576462 8682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:08.576500 8676 server_base.cc:1034] running on GCE node
I20260812 14:52:08.576865 8676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:08.577093 8676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:08.578231 8676 hybrid_clock.cc:648] HybridClock initialized: now 1786546328578216 us; error 28 us; skew 500 ppm
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:08.580515 8676 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:08.580890 8676 webserver.cc:460] Webserver started at http://127.4.27.1:42395/ using document root <none> and password file <none>
I20260812 14:52:08.581074 8676 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:08.581131 8676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:08.581257 8676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:08.582176 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "7666447f65624d528528b12ef0c1ed5f"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "02547952ef81aee6d7fe3009656de7d7"
server_key_iv: "86d47b583781a15284948c71442c831b"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.582530 8676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "7666447f65624d528528b12ef0c1ed5f"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "02547952ef81aee6d7fe3009656de7d7"
server_key_iv: "86d47b583781a15284948c71442c831b"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.583709 8676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:08.584349 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.584489 8676 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.584571 8676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "7666447f65624d528528b12ef0c1ed5f"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "02547952ef81aee6d7fe3009656de7d7"
server_key_iv: "86d47b583781a15284948c71442c831b"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.584631 8676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:08.607221 8676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:08.610636 8676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:08.610800 8676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:08.611047 8676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:08.611392 8676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:08.611443 8676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.611479 8676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:08.611502 8676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.617111 8676 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:34199
I20260812 14:52:08.617182 8899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:34199 every 8 connection(s)
I20260812 14:52:08.617471 8676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:08.624586 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8676
I20260812 14:52:08.624687 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:08.624871 4204 external_mini_cluster.cc:1428] Setting key 287e5378c5ab84ccfdd41a234f47cdfd
I20260812 14:52:08.625675 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.618130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41497 (local address 127.4.27.62:42273)
0812 14:52:08.618186 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:08.618188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:08.618361 (+ 173us) server_negotiation.cc:407] Connection header received
0812 14:52:08.618843 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:08.618844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:08.618856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:08.618910 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:08.619752 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.620241 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.621140 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.621340 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.621387 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.623214 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:08.623216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:08.623216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:08.623226 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:08.624352 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.624790 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.624791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.624791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.624844 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.625097 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.625098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.625098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.625231 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:08.625308 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:08.625539 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 14:52:08.625612 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:52:08.625876 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.618088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42273 (local address 127.4.27.1:41497)
0812 14:52:08.618236 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:08.618284 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:08.618766 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:08.618961 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:08.618964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:08.619499 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:08.619682 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:08.619686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.620308 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.620309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:08.621063 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:08.621065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.621397 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.621398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:08.621495 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.621861 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:08.621869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:08.623124 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:08.624422 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:08.624431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:08.624433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:08.624718 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:08.624897 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:08.624898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:08.624899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:08.625032 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:08.625306 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:08.625312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:08.625424 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 14:52:08.625512 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:52:08.625666 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260812 14:52:08.626751 8900 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42273
I20260812 14:52:08.626840 8900 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:08.627008 8900 heartbeater.cc:511] Master 127.4.27.62:42273 requested a full tablet report, sending...
I20260812 14:52:08.627521 8577 ts_manager.cc:194] Registered new tserver with Master: 7666447f65624d528528b12ef0c1ed5f (127.4.27.1:34199)
I20260812 14:52:08.628118 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:41497
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:08.646799 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42273
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:08.723906 8910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:08.724082 8910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:08.724112 8910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:08.725226 8910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:08.725265 8910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:08.725299 8910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:08.726693 8910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:08.726903 8910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:08.727095 8910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:08.731667 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.731671 8927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:08.731743 8910 server_base.cc:1034] running on GCE node
W20260812 14:52:08.731660 8925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:08.732074 8910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:08.732291 8910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:08.733479 8910 hybrid_clock.cc:648] HybridClock initialized: now 1786546328733466 us; error 28 us; skew 500 ppm
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:08.735878 8910 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:08.736280 8910 webserver.cc:460] Webserver started at http://127.4.27.2:34215/ using document root <none> and password file <none>
I20260812 14:52:08.736464 8910 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:08.736509 8910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:08.736639 8910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:08.737571 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "06d22b8a55d247a796a3bf63601ec95d"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "dda1eb15d4dc6486ca606edaca245eb2"
server_key_iv: "e81c805a0a22d08f8e0213cba3b5bf71"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.737891 8910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "06d22b8a55d247a796a3bf63601ec95d"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "dda1eb15d4dc6486ca606edaca245eb2"
server_key_iv: "e81c805a0a22d08f8e0213cba3b5bf71"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.739104 8910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:08.739887 8942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.740031 8910 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:08.740111 8910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "06d22b8a55d247a796a3bf63601ec95d"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "dda1eb15d4dc6486ca606edaca245eb2"
server_key_iv: "e81c805a0a22d08f8e0213cba3b5bf71"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.740168 8910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:08.756748 8910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:08.758719 8910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:08.758823 8910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:08.758991 8910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:08.759225 8910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:08.759254 8910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.759274 8910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:08.759289 8910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.764546 8910 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36661
I20260812 14:52:08.764585 9142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36661 every 8 connection(s)
I20260812 14:52:08.764844 8910 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:08.770764 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8910
I20260812 14:52:08.770859 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:08.771051 4204 external_mini_cluster.cc:1428] Setting key f78bc13ffef64eace04a44f0e00e7498
I20260812 14:52:08.773352 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.765461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:37731 (local address 127.4.27.62:42273)
0812 14:52:08.765525 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:08.765528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:08.765678 (+ 150us) server_negotiation.cc:407] Connection header received
0812 14:52:08.766146 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:08.766147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:08.766160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:08.766210 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:08.767095 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.767595 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.768525 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.768731 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.768789 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.770794 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:08.770795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:08.770796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:08.770802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:08.772083 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.772513 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.772513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.772513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.772559 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.772778 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.772778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.772778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.772868 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:08.772910 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:08.773188 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 14:52:08.773266 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:52:08.773532 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.765424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42273 (local address 127.4.27.2:37731)
0812 14:52:08.765574 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:08.765605 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:08.766049 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:08.766277 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:08.766281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:08.766834 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:08.767004 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:08.767009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.767673 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.767675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:08.768441 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:08.768443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.768800 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.768801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:08.768911 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.769328 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:08.769336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:08.770704 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:08.772156 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:08.772163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:08.772165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:08.772445 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:08.772605 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:08.772606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:08.772606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:08.772726 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:08.772916 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:08.772921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:08.773063 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 14:52:08.773164 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 14:52:08.773324 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260812 14:52:08.774020 9143 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42273
I20260812 14:52:08.774119 9143 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:08.774297 9143 heartbeater.cc:511] Master 127.4.27.62:42273 requested a full tablet report, sending...
I20260812 14:52:08.774669 8577 ts_manager.cc:194] Registered new tserver with Master: 06d22b8a55d247a796a3bf63601ec95d (127.4.27.2:36661)
I20260812 14:52:08.775025 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:37731
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:08.796104 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42273
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:08.874552 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:08.874728 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:08.874749 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:08.875860 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:08.875900 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:08.875923 9157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:08.877513 9157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:08.877720 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:08.877924 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:08.882629 9167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.882620 9171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:08.882622 9166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:08.882983 9157 server_base.cc:1034] running on GCE node
I20260812 14:52:08.883157 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:08.883342 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:08.884475 9157 hybrid_clock.cc:648] HybridClock initialized: now 1786546328884465 us; error 26 us; skew 500 ppm
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:08.886812 9157 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:08.887247 9157 webserver.cc:460] Webserver started at http://127.4.27.3:36627/ using document root <none> and password file <none>
I20260812 14:52:08.887449 9157 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:08.887491 9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:08.887610 9157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:08.888594 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "6813af11c8394ec0a9e48c9e76e65705"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "fd91fa73bc8460f8bf0c6db4c198953a"
server_key_iv: "67f868c6ac0826877cbd7e182a7b0e69"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.888907 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "6813af11c8394ec0a9e48c9e76e65705"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "fd91fa73bc8460f8bf0c6db4c198953a"
server_key_iv: "67f868c6ac0826877cbd7e182a7b0e69"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.890064 9157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:52:08.890758 9188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.890918 9157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.891002 9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "6813af11c8394ec0a9e48c9e76e65705"
format_stamp: "Formatted at 2026-08-12 14:52:08 on dist-test-slave-6zbq"
server_key: "fd91fa73bc8460f8bf0c6db4c198953a"
server_key_iv: "67f868c6ac0826877cbd7e182a7b0e69"
server_key_version: "encryptionkey@0"
I20260812 14:52:08.891053 9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:08.912678 9157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:08.914333 9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:08.914472 9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:08.914652 9157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:08.914891 9157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:08.914921 9157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.914940 9157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:08.914951 9157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:08.920069 9157 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:44161
I20260812 14:52:08.920133 9391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:44161 every 8 connection(s)
I20260812 14:52:08.920388 9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:52:08 dist-test-slave-6zbq krb5kdc[8489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546328, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:08.928155 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.921006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:59529 (local address 127.4.27.62:42273)
0812 14:52:08.921065 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:08.921067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:08.921256 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:52:08.921698 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:08.921699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:08.921709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:08.921762 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:08.922591 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.923073 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.923970 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.924142 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.924184 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.926065 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:08.926066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:08.926067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:08.926073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:08.927164 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.927495 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.927496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.927496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.927538 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:08.927715 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:08.927715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:08.927715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:08.927810 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:08.927853 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:08.928042 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:52:08.928095 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:52:08.928320 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:08.920983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42273 (local address 127.4.27.3:59529)
0812 14:52:08.921149 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:08.921183 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:08.921626 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:08.921813 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:08.921816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:08.922333 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:08.922524 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:08.922529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.923134 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.923134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:08.923908 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:08.923911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:08.924195 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:08.924196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:08.924287 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:08.924724 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:08.924734 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:08.925976 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:08.927231 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:08.927237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:08.927239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:08.927441 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:08.927580 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:08.927581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:08.927581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:08.927674 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:08.927851 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:08.927853 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:08.927952 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 14:52:08.928023 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:52:08.928138 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260812 14:52:08.928745 9394 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42273
I20260812 14:52:08.928830 9394 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:08.928985 9394 heartbeater.cc:511] Master 127.4.27.62:42273 requested a full tablet report, sending...
I20260812 14:52:08.929353 8577 ts_manager.cc:194] Registered new tserver with Master: 6813af11c8394ec0a9e48c9e76e65705 (127.4.27.3:44161)
I20260812 14:52:08.929684 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:59529
I20260812 14:52:08.930583 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9157
I20260812 14:52:08.930661 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:08.930816 4204 external_mini_cluster.cc:1428] Setting key d7bbd05996ae4ad29526479eebb2bf10
I20260812 14:52:08.931566 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:52:08.942629 8638 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:08.941867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.4.27.62:42273)
0812 14:52:08.941925 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:08.941926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:08.942018 (+ 92us) server_negotiation.cc:407] Connection header received
0812 14:52:08.942062 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:08.942062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:08.942070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:08.942114 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:08.942578 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33558: BlockingRecv error: recv got EOF from 127.0.0.1:33558 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:52:08.943481 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8676
I20260812 14:52:08.966913 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8910
I20260812 14:52:08.974977 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9157
I20260812 14:52:08.986809 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8520
2026-08-12T14:52:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1963 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:09 dist-test-slave-6zbq krb5kdc[9412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:09 dist-test-slave-6zbq krb5kdc[9412](info): set up 2 sockets
Aug 12 14:52:09 dist-test-slave-6zbq krb5kdc[9412](info): commencing operation
krb5kdc: starting...
W20260812 14:52:10.169756 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:10 dist-test-slave-6zbq krb5kdc[9412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:10.274868 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41471
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:10.352596 9428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:10.352779 9428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:10.352799 9428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:10.353857 9428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:10.353889 9428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:10.353901 9428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:10.353912 9428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:10.353924 9428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:10.355347 9428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:10.355520 9428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:10.355679 9428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:10.359941 9445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:10.360347 9437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:10.360355 9439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:10.360380 9428 server_base.cc:1034] running on GCE node
I20260812 14:52:10.360669 9428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:10.360894 9428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:10.362056 9428 hybrid_clock.cc:648] HybridClock initialized: now 1786546330361978 us; error 95 us; skew 500 ppm
Aug 12 14:52:10 dist-test-slave-6zbq krb5kdc[9412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546330, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:10.369155 9428 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:10.369665 9428 webserver.cc:460] Webserver started at http://127.4.27.62:38599/ using document root <none> and password file <none>
I20260812 14:52:10.369882 9428 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:10.369975 9428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:10.370110 9428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:10.371455 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "c208d9dcef044b49af31624ed90cf810"
format_stamp: "Formatted at 2026-08-12 14:52:10 on dist-test-slave-6zbq"
server_key: "cb53611d8c3886021fa571fb1526ff42"
server_key_iv: "50d68f680622179f34facba76343d59a"
server_key_version: "encryptionkey@0"
I20260812 14:52:10.371861 9428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "c208d9dcef044b49af31624ed90cf810"
format_stamp: "Formatted at 2026-08-12 14:52:10 on dist-test-slave-6zbq"
server_key: "cb53611d8c3886021fa571fb1526ff42"
server_key_iv: "50d68f680622179f34facba76343d59a"
server_key_version: "encryptionkey@0"
I20260812 14:52:10.373116 9428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:10.373718 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:10.373860 9428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:10.373945 9428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "c208d9dcef044b49af31624ed90cf810"
format_stamp: "Formatted at 2026-08-12 14:52:10 on dist-test-slave-6zbq"
server_key: "cb53611d8c3886021fa571fb1526ff42"
server_key_iv: "50d68f680622179f34facba76343d59a"
server_key_version: "encryptionkey@0"
I20260812 14:52:10.374012 9428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:10.391458 9428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:10.392968 9428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:10.393091 9428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:10.396728 9428 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:41471
I20260812 14:52:10.396768 9548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:41471 every 8 connection(s)
I20260812 14:52:10.397053 9428 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:10.397847 9549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:10.398881 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9428
I20260812 14:52:10.398979 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:10.399155 4204 external_mini_cluster.cc:1428] Setting key e1794b37a612ac28358f5bd13f0cd568
I20260812 14:52:10.400513 9549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810: Bootstrap starting.
I20260812 14:52:10.401245 9549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:10.401520 9549 log.cc:824] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:10.402136 9549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810: No bootstrap required, opened a new log
I20260812 14:52:10.404351 9549 raft_consensus.cc:348] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c208d9dcef044b49af31624ed90cf810" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41471 } }
I20260812 14:52:10.404516 9549 raft_consensus.cc:374] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:10.404565 9549 raft_consensus.cc:729] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c208d9dcef044b49af31624ed90cf810, State: Initialized, Role: FOLLOWER
I20260812 14:52:10.404695 9549 consensus_queue.cc:260] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c208d9dcef044b49af31624ed90cf810" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41471 } }
I20260812 14:52:10.404773 9549 raft_consensus.cc:388] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:10.404811 9549 raft_consensus.cc:482] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:10.404858 9549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:10.405426 9549 raft_consensus.cc:504] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c208d9dcef044b49af31624ed90cf810" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41471 } }
I20260812 14:52:10.405514 9549 leader_election.cc:302] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c208d9dcef044b49af31624ed90cf810; no voters:
I20260812 14:52:10.405676 9549 leader_election.cc:288] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [CANDIDATE]: Term 1 election: Requested vote from peers
Aug 12 14:52:10 dist-test-slave-6zbq krb5kdc[9412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:10.405725 9557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:10.405877 9557 raft_consensus.cc:686] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [term 1 LEADER]: Becoming Leader. State: Replica: c208d9dcef044b49af31624ed90cf810, State: Running, Role: LEADER
I20260812 14:52:10.405928 9549 sys_catalog.cc:564] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:10.406036 9557 consensus_queue.cc:237] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c208d9dcef044b49af31624ed90cf810" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41471 } }
I20260812 14:52:10.406394 9557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c208d9dcef044b49af31624ed90cf810. Latest consensus state: current_term: 1 leader_uuid: "c208d9dcef044b49af31624ed90cf810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c208d9dcef044b49af31624ed90cf810" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41471 } } }
I20260812 14:52:10.406404 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c208d9dcef044b49af31624ed90cf810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c208d9dcef044b49af31624ed90cf810" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41471 } } }
I20260812 14:52:10.406562 9557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:10.406572 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:10.406850 9575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:10.407401 9575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:10.409032 9575 catalog_manager.cc:1347] Generated new cluster ID: fb874c3c16f645e88771bd3bd3742aaf
I20260812 14:52:10.409086 9575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:10.409674 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:10.399644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.4.27.62:41471)
0812 14:52:10.399814 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:10.399820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:10.399844 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:52:10.400320 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:10.400327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:10.400371 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:10.400480 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:10.401240 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:10.402277 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:10.403290 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:10.403664 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:10.403737 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:10.406258 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:10.406262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:10.406263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:10.406272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:10.408062 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:10.408450 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:10.408453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:10.408453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:10.408550 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:10.408740 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:10.408742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:10.408742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:10.409031 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:10.409208 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:10.409350 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 14:52:10.409487 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260812 14:52:10.410686 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:52:10.433493 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:10.429372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.4.27.62:41471)
0812 14:52:10.429412 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:10.429414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:10.429502 (+ 88us) server_negotiation.cc:407] Connection header received
0812 14:52:10.429568 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:10.429569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:10.429579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:10.429632 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:10.430024 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:10.430794 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:10.431914 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:10.432248 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:10.432315 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:10.433007 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:10.433009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:10.433009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:10.433237 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:10.433257 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:10.433265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:10.433320 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:10.433386 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 14:52:10.433413 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:10.435657 9575 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:10.436129 9575 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:10.441821 9575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c208d9dcef044b49af31624ed90cf810: Generated new TSK 0
I20260812 14:52:10.441977 9575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:10.447395 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9428
2026-08-12T14:52:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1454 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:10 dist-test-slave-6zbq krb5kdc[9609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:10 dist-test-slave-6zbq krb5kdc[9609](info): set up 2 sockets
Aug 12 14:52:10 dist-test-slave-6zbq krb5kdc[9609](info): commencing operation
krb5kdc: starting...
W20260812 14:52:11.625550 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:11 dist-test-slave-6zbq krb5kdc[9609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:11.732921 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46153
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:11.810967 9638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:11.811141 9638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:11.811162 9638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:11.812261 9638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:11.812300 9638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:11.812314 9638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:11.812335 9638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:11.812359 9638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:11.813760 9638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:11.813964 9638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:11.814160 9638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:11.818552 9651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:11.818542 9649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:11.818552 9647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:11.818569 9638 server_base.cc:1034] running on GCE node
I20260812 14:52:11.818979 9638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:11.819232 9638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:11.820369 9638 hybrid_clock.cc:648] HybridClock initialized: now 1786546331820354 us; error 31 us; skew 500 ppm
Aug 12 14:52:11 dist-test-slave-6zbq krb5kdc[9609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:11.822618 9638 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:11.823032 9638 webserver.cc:460] Webserver started at http://127.4.27.62:40123/ using document root <none> and password file <none>
I20260812 14:52:11.823235 9638 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:11.823282 9638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:11.823398 9638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:11.824362 9638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "54cf1f1ba6694ab2a07fe3d050586e80"
format_stamp: "Formatted at 2026-08-12 14:52:11 on dist-test-slave-6zbq"
server_key: "d4085620de711f77dc9424698afce8eb"
server_key_iv: "6a106034d76327d7109744b2b4e4194c"
server_key_version: "encryptionkey@0"
I20260812 14:52:11.824673 9638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "54cf1f1ba6694ab2a07fe3d050586e80"
format_stamp: "Formatted at 2026-08-12 14:52:11 on dist-test-slave-6zbq"
server_key: "d4085620de711f77dc9424698afce8eb"
server_key_iv: "6a106034d76327d7109744b2b4e4194c"
server_key_version: "encryptionkey@0"
I20260812 14:52:11.825878 9638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:52:11.826576 9664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:11.826715 9638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:11.826800 9638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "54cf1f1ba6694ab2a07fe3d050586e80"
format_stamp: "Formatted at 2026-08-12 14:52:11 on dist-test-slave-6zbq"
server_key: "d4085620de711f77dc9424698afce8eb"
server_key_iv: "6a106034d76327d7109744b2b4e4194c"
server_key_version: "encryptionkey@0"
I20260812 14:52:11.826858 9638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:11.846041 9638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:11.847636 9638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:11.847762 9638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:11.851313 9638 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46153
I20260812 14:52:11.851344 9744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46153 every 8 connection(s)
I20260812 14:52:11.851629 9638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:11.852622 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:11.854797 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80: Bootstrap starting.
I20260812 14:52:11.855300 9745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:11.855522 9745 log.cc:824] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:11.856102 9745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80: No bootstrap required, opened a new log
I20260812 14:52:11.857017 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9638
I20260812 14:52:11.857107 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:11.857282 4204 external_mini_cluster.cc:1428] Setting key fe227c0af45b355df6be0e43a0d6c2c1
I20260812 14:52:11.857965 9745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46153 } }
I20260812 14:52:11.858095 9745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:11.858116 9745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cf1f1ba6694ab2a07fe3d050586e80, State: Initialized, Role: FOLLOWER
I20260812 14:52:11.858224 9745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46153 } }
I20260812 14:52:11.858295 9745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:11.858325 9745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:11.858415 9745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:11.858896 9745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46153 } }
I20260812 14:52:11.858990 9745 leader_election.cc:302] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cf1f1ba6694ab2a07fe3d050586e80; no voters:
I20260812 14:52:11.859138 9745 leader_election.cc:288] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:11.859177 9752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:11.859364 9752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [term 1 LEADER]: Becoming Leader. State: Replica: 54cf1f1ba6694ab2a07fe3d050586e80, State: Running, Role: LEADER
I20260812 14:52:11.859396 9745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:11.859481 9752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46153 } }
I20260812 14:52:11.859820 9752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cf1f1ba6694ab2a07fe3d050586e80. Latest consensus state: current_term: 1 leader_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46153 } } }
I20260812 14:52:11.859894 9752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:11.859870 9754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cf1f1ba6694ab2a07fe3d050586e80" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46153 } } }
I20260812 14:52:11.859944 9754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:11.860618 9762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:11.861167 9762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:11.862663 9762 catalog_manager.cc:1347] Generated new cluster ID: 17e48962ffee4017a522775b75325d54
I20260812 14:52:11.862715 9762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:52:11 dist-test-slave-6zbq krb5kdc[9609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:11.866925 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:11.857758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.4.27.62:46153)
0812 14:52:11.857920 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:11.857928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:11.857954 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:52:11.858039 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:11.858044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:11.858083 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:11.858228 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:11.858930 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:11.860168 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:11.861131 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:11.861580 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:11.861675 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:11.863965 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:11.863969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:11.863970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:11.863981 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:11.865438 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:11.865751 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:11.865754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:11.865755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:11.865847 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:11.866024 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:11.866025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:11.866025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:11.866252 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:11.866414 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:11.866521 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 14:52:11.866652 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260812 14:52:11.870447 9762 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:11.870954 9762 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:11.873195 9762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54cf1f1ba6694ab2a07fe3d050586e80: Generated new TSK 0
I20260812 14:52:11.873337 9762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:11.902607 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46153
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:12.003760 9794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:12.003933 9794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:12.003954 9794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:12.005059 9794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:12.005098 9794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:12.005122 9794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:12.006523 9794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:12.006723 9794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:12.006929 9794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:52:12.011461 9794 server_base.cc:1034] running on GCE node
W20260812 14:52:12.011446 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:12.011454 9805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:12.011438 9804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:12.011830 9794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:12.012054 9794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:12.013248 9794 hybrid_clock.cc:648] HybridClock initialized: now 1786546332013230 us; error 37 us; skew 500 ppm
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:12.015686 9794 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:12.016105 9794 webserver.cc:460] Webserver started at http://127.4.27.1:40179/ using document root <none> and password file <none>
I20260812 14:52:12.016327 9794 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:12.016376 9794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:12.016496 9794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:12.017504 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "70df23b06224496785d5de6cbe566344"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "56962547ee2781d7d499d944a6790101"
server_key_iv: "a521ff72248513d10871aba120ce6b7b"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.017818 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "70df23b06224496785d5de6cbe566344"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "56962547ee2781d7d499d944a6790101"
server_key_iv: "a521ff72248513d10871aba120ce6b7b"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.019093 9794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:12.019738 9823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.019882 9794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:12.019964 9794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "70df23b06224496785d5de6cbe566344"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "56962547ee2781d7d499d944a6790101"
server_key_iv: "a521ff72248513d10871aba120ce6b7b"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.020023 9794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:12.042079 9794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:12.043699 9794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:12.043829 9794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:12.044016 9794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:12.044288 9794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:12.044320 9794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.044348 9794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:12.044364 9794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.049856 9794 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:39525
I20260812 14:52:12.049891 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:39525 every 8 connection(s)
I20260812 14:52:12.050163 9794 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:12.057615 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9794
I20260812 14:52:12.057706 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:12.057873 4204 external_mini_cluster.cc:1428] Setting key 7cbc0f6dc40dabfdfeb3f36e8c532b2b
I20260812 14:52:12.058724 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.050809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:48373 (local address 127.4.27.62:46153)
0812 14:52:12.050853 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:12.050855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:12.051048 (+ 193us) server_negotiation.cc:407] Connection header received
0812 14:52:12.051514 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:12.051515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:12.051529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:12.051574 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:12.052447 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.052944 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.053944 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.054151 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.054213 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.056159 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:12.056161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:12.056161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:12.056169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:12.057274 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.057631 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.057632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.057633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.057686 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.057932 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.057933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.057933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.058072 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:12.058149 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:12.058545 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 14:52:12.058635 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:52:12.058930 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.050784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46153 (local address 127.4.27.1:48373)
0812 14:52:12.050945 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:12.050978 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:12.051425 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:12.051650 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:12.051654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:12.052179 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:12.052354 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:12.052358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.053025 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.053026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:12.053863 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:12.053866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.054222 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.054223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:12.054365 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.054779 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:12.054788 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:12.056065 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:12.057347 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:12.057354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:12.057356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:12.057562 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:12.057758 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:12.057759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:12.057759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:12.057864 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:12.058147 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:12.058154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:12.058281 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:52:12.058408 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 14:52:12.058573 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260812 14:52:12.059365 10010 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46153
I20260812 14:52:12.059448 10010 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:12.059603 10010 heartbeater.cc:511] Master 127.4.27.62:46153 requested a full tablet report, sending...
I20260812 14:52:12.060156 9687 ts_manager.cc:194] Registered new tserver with Master: 70df23b06224496785d5de6cbe566344 (127.4.27.1:39525)
I20260812 14:52:12.060740 9687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:48373
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:12.081418 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46153
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:12.167264 10021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:12.167505 10021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:12.167578 10021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:12.169157 10021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:12.169207 10021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:12.169242 10021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:12.170914 10021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:12.171093 10021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:12.171268 10021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:12.175890 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:12.175899 10040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:12.175961 10021 server_base.cc:1034] running on GCE node
W20260812 14:52:12.175902 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:12.176278 10021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:12.176483 10021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:12.177609 10021 hybrid_clock.cc:648] HybridClock initialized: now 1786546332177597 us; error 28 us; skew 500 ppm
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:12.179930 10021 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:12.180311 10021 webserver.cc:460] Webserver started at http://127.4.27.2:40479/ using document root <none> and password file <none>
I20260812 14:52:12.180488 10021 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:12.180529 10021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:12.180650 10021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:12.181582 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "fa53d8d0cfc94924a3ec9be058e5f294"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "8c7ff2bda9dc94d5854a1623c06f0e22"
server_key_iv: "77c203eb6cdb1ff3ff5914a703d8e5a1"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.181900 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "fa53d8d0cfc94924a3ec9be058e5f294"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "8c7ff2bda9dc94d5854a1623c06f0e22"
server_key_iv: "77c203eb6cdb1ff3ff5914a703d8e5a1"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.183104 10021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:12.183701 10052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.183853 10021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:12.183935 10021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "fa53d8d0cfc94924a3ec9be058e5f294"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "8c7ff2bda9dc94d5854a1623c06f0e22"
server_key_iv: "77c203eb6cdb1ff3ff5914a703d8e5a1"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.183986 10021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:12.197705 10021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:12.199589 10021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:12.199689 10021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:12.199848 10021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:12.200070 10021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:12.200099 10021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.200117 10021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:12.200134 10021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.205142 10021 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:41343
I20260812 14:52:12.205204 10241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:41343 every 8 connection(s)
I20260812 14:52:12.205438 10021 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:12.209277 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10021
I20260812 14:52:12.209383 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:12.209568 4204 external_mini_cluster.cc:1428] Setting key a655d89783f6beffaf603c09ea452408
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:12.213919 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.206033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:41067 (local address 127.4.27.62:46153)
0812 14:52:12.206076 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:12.206077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:12.206249 (+ 172us) server_negotiation.cc:407] Connection header received
0812 14:52:12.206706 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:12.206707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:12.206715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:12.206762 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:12.207559 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.208041 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.208925 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.209118 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.209173 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.211330 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:12.211332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:12.211332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:12.211338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:12.212698 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.213085 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.213087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.213087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.213143 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.213382 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.213383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.213383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.213475 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:12.213518 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:12.213795 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 14:52:12.213859 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:52:12.214120 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.205982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46153 (local address 127.4.27.2:41067)
0812 14:52:12.206137 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:12.206173 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:12.206629 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:12.206818 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:12.206821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:12.207327 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:12.207493 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:12.207498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.208099 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.208100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:12.208849 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:12.208851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.209209 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.209211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:12.209354 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.209788 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:12.209795 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:12.211240 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:12.212786 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:12.212791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:12.212793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:12.213014 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:12.213198 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:12.213199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:12.213199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:12.213324 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:12.213523 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:12.213528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:12.213667 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 14:52:12.213784 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 14:52:12.213929 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260812 14:52:12.214564 10247 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46153
I20260812 14:52:12.214660 10247 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:12.214841 10247 heartbeater.cc:511] Master 127.4.27.62:46153 requested a full tablet report, sending...
I20260812 14:52:12.215170 9687 ts_manager.cc:194] Registered new tserver with Master: fa53d8d0cfc94924a3ec9be058e5f294 (127.4.27.2:41343)
I20260812 14:52:12.215512 9687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:41067
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:12.233171 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46153
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:12.312752 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:12.312943 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:12.312973 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:12.314076 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:12.314116 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:12.314149 10252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:12.315555 10252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:12.315781 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:12.315976 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:12.320448 10265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:12.320457 10272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:12.320457 10267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:12.320631 10252 server_base.cc:1034] running on GCE node
I20260812 14:52:12.320811 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:12.321022 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:12.322151 10252 hybrid_clock.cc:648] HybridClock initialized: now 1786546332322140 us; error 34 us; skew 500 ppm
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:12.324430 10252 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:12.324785 10252 webserver.cc:460] Webserver started at http://127.4.27.3:42845/ using document root <none> and password file <none>
I20260812 14:52:12.324965 10252 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:12.325003 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:12.325091 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:12.325987 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "825cf65e8b95487b8dcd7bb0305e069c"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "349d17d7aa7c03ad9f5461bed254086a"
server_key_iv: "1a9b2aa48f38d778438e78ce7097ab30"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.326275 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "825cf65e8b95487b8dcd7bb0305e069c"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "349d17d7aa7c03ad9f5461bed254086a"
server_key_iv: "1a9b2aa48f38d778438e78ce7097ab30"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.327554 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:12.328158 10286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.328306 10252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:12.328384 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "825cf65e8b95487b8dcd7bb0305e069c"
format_stamp: "Formatted at 2026-08-12 14:52:12 on dist-test-slave-6zbq"
server_key: "349d17d7aa7c03ad9f5461bed254086a"
server_key_iv: "1a9b2aa48f38d778438e78ce7097ab30"
server_key_version: "encryptionkey@0"
I20260812 14:52:12.328438 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:12.337738 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:12.339303 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:12.339398 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:12.339547 10252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:12.339767 10252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:12.339795 10252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.339813 10252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:12.339829 10252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:12.344940 10252 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42679
I20260812 14:52:12.344978 10494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42679 every 8 connection(s)
I20260812 14:52:12.345225 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:12.347154 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10252
I20260812 14:52:12.347226 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:12.347379 4204 external_mini_cluster.cc:1428] Setting key 1eb73dfd80562987b57e4b94f87e2240
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:12.353363 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.345826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:59131 (local address 127.4.27.62:46153)
0812 14:52:12.345867 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:12.345869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:12.346043 (+ 174us) server_negotiation.cc:407] Connection header received
0812 14:52:12.346546 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:12.346547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:12.346559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:12.346606 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:12.347500 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.348138 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.349056 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.349263 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.349317 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.351153 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:12.351155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:12.351155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:12.351163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:12.352174 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.352515 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.352516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.352516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.352574 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.352786 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.352786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.352787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.352895 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:12.352950 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:12.353190 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 14:52:12.353277 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:12.353487 10499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.345807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46153 (local address 127.4.27.3:59131)
0812 14:52:12.345941 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:12.345971 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:12.346446 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:12.346668 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:12.346672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:12.347206 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:12.347418 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:12.347424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.348218 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.348219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:12.348978 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:12.348981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.349331 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.349332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:12.349453 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.349854 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:12.349862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:12.351071 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:12.352244 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:12.352251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:12.352253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:12.352452 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:12.352628 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:12.352629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:12.352629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:12.352731 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:12.352948 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:12.352955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:12.353075 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 14:52:12.353159 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:52:12.353299 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260812 14:52:12.353915 10496 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46153
I20260812 14:52:12.353991 10496 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:12.354116 10496 heartbeater.cc:511] Master 127.4.27.62:46153 requested a full tablet report, sending...
I20260812 14:52:12.354425 9687 ts_manager.cc:194] Registered new tserver with Master: 825cf65e8b95487b8dcd7bb0305e069c (127.4.27.3:42679)
I20260812 14:52:12.354770 9687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:59131
I20260812 14:52:12.358909 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[9609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546332, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:12.381100 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:12.374470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.4.27.62:46153)
0812 14:52:12.374508 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:12.374511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:12.374627 (+ 116us) server_negotiation.cc:407] Connection header received
0812 14:52:12.374671 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:12.374672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:12.374680 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:12.374724 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:12.375159 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.375599 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.376381 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:12.376679 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:12.376736 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:12.379076 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:12.379077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:12.379077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:12.379083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:12.380267 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.380548 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.380548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.380548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.380590 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:12.380731 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:12.380732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:12.380732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:12.380846 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:12.380914 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:12.380978 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 14:52:12.381032 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:52:12.381312 9687 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48396
I20260812 14:52:12.381664 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9794
I20260812 14:52:12.394454 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10021
I20260812 14:52:12.407791 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10252
I20260812 14:52:12.419281 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9638
2026-08-12T14:52:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1973 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[10516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[10516](info): set up 2 sockets
Aug 12 14:52:12 dist-test-slave-6zbq krb5kdc[10516](info): commencing operation
krb5kdc: starting...
W20260812 14:52:13.588917 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:13 dist-test-slave-6zbq krb5kdc[10516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:13.708639 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42497
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:42497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:13.787882 10536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:13.788056 10536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:13.788076 10536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:13.789163 10536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:13.789202 10536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:13.789215 10536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:13.789248 10536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:13.789270 10536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:13.790791 10536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:42497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:13.791005 10536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:13.791188 10536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:13.795814 10547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:13.795863 10536 server_base.cc:1034] running on GCE node
W20260812 14:52:13.795832 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:13.795832 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:13.796255 10536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:13.796530 10536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:13.797667 10536 hybrid_clock.cc:648] HybridClock initialized: now 1786546333797652 us; error 29 us; skew 500 ppm
Aug 12 14:52:13 dist-test-slave-6zbq krb5kdc[10516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:13.799925 10536 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:13.800359 10536 webserver.cc:460] Webserver started at http://127.4.27.62:46677/ using document root <none> and password file <none>
I20260812 14:52:13.800562 10536 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:13.800609 10536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:13.800733 10536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:13.801714 10536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "ffa780fdf00d491ba4f602b51082038d"
format_stamp: "Formatted at 2026-08-12 14:52:13 on dist-test-slave-6zbq"
server_key: "8809eae055fc2f66c939e0529140591c"
server_key_iv: "818339266f30b2e851fa607975839897"
server_key_version: "encryptionkey@0"
I20260812 14:52:13.802031 10536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "ffa780fdf00d491ba4f602b51082038d"
format_stamp: "Formatted at 2026-08-12 14:52:13 on dist-test-slave-6zbq"
server_key: "8809eae055fc2f66c939e0529140591c"
server_key_iv: "818339266f30b2e851fa607975839897"
server_key_version: "encryptionkey@0"
I20260812 14:52:13.803301 10536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:13.803925 10566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:13.804073 10536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:13.804157 10536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "ffa780fdf00d491ba4f602b51082038d"
format_stamp: "Formatted at 2026-08-12 14:52:13 on dist-test-slave-6zbq"
server_key: "8809eae055fc2f66c939e0529140591c"
server_key_iv: "818339266f30b2e851fa607975839897"
server_key_version: "encryptionkey@0"
I20260812 14:52:13.804231 10536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:13.824246 10536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:13.826932 10536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:13.827077 10536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:13.831019 10536 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:42497
I20260812 14:52:13.831051 10664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:42497 every 8 connection(s)
I20260812 14:52:13.831602 10536 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:13.832244 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:13.832666 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10536
I20260812 14:52:13.832770 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:13.832940 4204 external_mini_cluster.cc:1428] Setting key a223c0ca7fd6054ce313ca78bb6a7336
I20260812 14:52:13.835043 10665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d: Bootstrap starting.
I20260812 14:52:13.835755 10665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:13.836048 10665 log.cc:824] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:13.836730 10665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d: No bootstrap required, opened a new log
I20260812 14:52:13.838950 10665 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa780fdf00d491ba4f602b51082038d" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42497 } }
I20260812 14:52:13.839097 10665 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:13.839131 10665 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa780fdf00d491ba4f602b51082038d, State: Initialized, Role: FOLLOWER
I20260812 14:52:13.839268 10665 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa780fdf00d491ba4f602b51082038d" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42497 } }
I20260812 14:52:13.839350 10665 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:13.839385 10665 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:13.839430 10665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:52:13 dist-test-slave-6zbq krb5kdc[10516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:13.840014 10665 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa780fdf00d491ba4f602b51082038d" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42497 } }
I20260812 14:52:13.840108 10665 leader_election.cc:302] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa780fdf00d491ba4f602b51082038d; no voters:
I20260812 14:52:13.840276 10665 leader_election.cc:288] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:13.840348 10672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:13.840507 10672 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [term 1 LEADER]: Becoming Leader. State: Replica: ffa780fdf00d491ba4f602b51082038d, State: Running, Role: LEADER
I20260812 14:52:13.840533 10665 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:13.840611 10672 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa780fdf00d491ba4f602b51082038d" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42497 } }
I20260812 14:52:13.840926 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa780fdf00d491ba4f602b51082038d. Latest consensus state: current_term: 1 leader_uuid: "ffa780fdf00d491ba4f602b51082038d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa780fdf00d491ba4f602b51082038d" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42497 } } }
I20260812 14:52:13.840937 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa780fdf00d491ba4f602b51082038d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa780fdf00d491ba4f602b51082038d" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42497 } } }
I20260812 14:52:13.841006 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:13.841012 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d [sys.catalog]: This master's current role is: LEADER
W20260812 14:52:13.842540 10698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:52:13.842586 10698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:52:13.842638 10684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:13.842770 10684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:13.843919 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:13.833457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.4.27.62:42497)
0812 14:52:13.833636 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:13.833647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:13.833692 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:52:13.834206 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:13.834216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:13.834277 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:13.834444 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:13.835022 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:13.836324 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:13.837086 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:13.837680 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:13.837801 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:13.840155 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:13.840160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:13.840161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:13.840172 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:13.842217 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:13.842640 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:13.842643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:13.842643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:13.842756 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:13.843016 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:13.843016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:13.843017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:13.843306 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:13.843434 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:13.843598 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 14:52:13.843736 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260812 14:52:13.844305 10684 catalog_manager.cc:1347] Generated new cluster ID: 432752c32131489dbba64eb0d35bf89a
I20260812 14:52:13.844355 10684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:13.870302 10684 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:13.871018 10684 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:13.881608 10684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa780fdf00d491ba4f602b51082038d: Generated new TSK 0
I20260812 14:52:13.881778 10684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:13.905625 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42497
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:13.984503 10703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:13.984685 10703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:13.984706 10703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:13.985759 10703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:13.985792 10703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:13.985816 10703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:13.987298 10703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:13.987483 10703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:13.987643 10703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:13.992290 10708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:13.992350 10703 server_base.cc:1034] running on GCE node
W20260812 14:52:13.992434 10710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:13.992303 10712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:13.992657 10703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:13.992875 10703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:13.994007 10703 hybrid_clock.cc:648] HybridClock initialized: now 1786546333993996 us; error 30 us; skew 500 ppm
Aug 12 14:52:13 dist-test-slave-6zbq krb5kdc[10516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:13.996224 10703 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:13.996593 10703 webserver.cc:460] Webserver started at http://127.4.27.1:45089/ using document root <none> and password file <none>
I20260812 14:52:13.996796 10703 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:13.996843 10703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:13.996960 10703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:13.997889 10703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "53748b3be59c47f3a742fa74041a7266"
format_stamp: "Formatted at 2026-08-12 14:52:13 on dist-test-slave-6zbq"
server_key: "374a9f42dcc10bebbe45bbc9e0e108a0"
server_key_iv: "a12dfb0d84885e8a1e602a14106fd408"
server_key_version: "encryptionkey@0"
I20260812 14:52:13.998196 10703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "53748b3be59c47f3a742fa74041a7266"
format_stamp: "Formatted at 2026-08-12 14:52:13 on dist-test-slave-6zbq"
server_key: "374a9f42dcc10bebbe45bbc9e0e108a0"
server_key_iv: "a12dfb0d84885e8a1e602a14106fd408"
server_key_version: "encryptionkey@0"
I20260812 14:52:13.999410 10703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:14.000057 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.000205 10703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:14.000288 10703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "53748b3be59c47f3a742fa74041a7266"
format_stamp: "Formatted at 2026-08-12 14:52:13 on dist-test-slave-6zbq"
server_key: "374a9f42dcc10bebbe45bbc9e0e108a0"
server_key_iv: "a12dfb0d84885e8a1e602a14106fd408"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.000351 10703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:14.012607 10703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:14.014138 10703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:14.014256 10703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:14.014463 10703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:14.014729 10703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:14.014761 10703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.014786 10703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:14.014804 10703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.020071 10703 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:42497
I20260812 14:52:14.020104 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:42497 every 8 connection(s)
I20260812 14:52:14.020358 10703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546333, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:14.028085 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.020962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:53559 (local address 127.4.27.62:42497)
0812 14:52:14.021024 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:14.021026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:14.021213 (+ 187us) server_negotiation.cc:407] Connection header received
0812 14:52:14.021989 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:14.021990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:14.021999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:14.022044 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:14.022603 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.023080 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.023966 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.024152 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.024195 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.026016 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:14.026017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:14.026017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:14.026024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:14.027126 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.027444 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.027445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.027446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.027487 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.027651 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.027651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.027651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.027738 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:14.027780 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:14.027962 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 14:52:14.028025 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:52:14.028209 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.020940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42497 (local address 127.4.27.1:53559)
0812 14:52:14.021099 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:14.021136 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:14.021913 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:14.022098 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:14.022101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:14.022316 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:14.022532 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:14.022537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.023142 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.023142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:14.023909 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:14.023911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.024204 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.024204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:14.024290 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.024713 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:14.024721 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:14.025934 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:14.027189 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:14.027194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:14.027196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:14.027390 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:14.027529 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:14.027530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:14.027530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:14.027612 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:14.027779 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:14.027781 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:14.027878 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 14:52:14.027947 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:52:14.028050 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260812 14:52:14.028620 10919 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42497
I20260812 14:52:14.028704 10919 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:14.028831 10919 heartbeater.cc:511] Master 127.4.27.62:42497 requested a full tablet report, sending...
I20260812 14:52:14.029273 10595 ts_manager.cc:194] Registered new tserver with Master: 53748b3be59c47f3a742fa74041a7266 (127.4.27.1:42497)
I20260812 14:52:14.029840 10595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:53559
I20260812 14:52:14.029870 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10703
I20260812 14:52:14.029951 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:14.030115 4204 external_mini_cluster.cc:1428] Setting key 1d60b568f6eb21c1946f91e3cacb228a
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:14.051106 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42497
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:14.129325 10931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:14.129506 10931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:14.129528 10931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:14.130636 10931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:14.130672 10931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:14.130697 10931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:14.132144 10931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:14.132318 10931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:14.132479 10931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:52:14.136996 10931 server_base.cc:1034] running on GCE node
W20260812 14:52:14.137029 10941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:14.136957 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:14.136977 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:14.137360 10931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:14.137548 10931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:14.138674 10931 hybrid_clock.cc:648] HybridClock initialized: now 1786546334138661 us; error 26 us; skew 500 ppm
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:14.141036 10931 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:14.141462 10931 webserver.cc:460] Webserver started at http://127.4.27.2:40749/ using document root <none> and password file <none>
I20260812 14:52:14.141672 10931 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:14.141719 10931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:14.141842 10931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:14.142868 10931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "ba2f3a5913414e2397bb87d44ce3f226"
format_stamp: "Formatted at 2026-08-12 14:52:14 on dist-test-slave-6zbq"
server_key: "d863cdbf7cf342c66677e84d198c24c4"
server_key_iv: "e81957e00a0553b9511eed62cef2412f"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.143195 10931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "ba2f3a5913414e2397bb87d44ce3f226"
format_stamp: "Formatted at 2026-08-12 14:52:14 on dist-test-slave-6zbq"
server_key: "d863cdbf7cf342c66677e84d198c24c4"
server_key_iv: "e81957e00a0553b9511eed62cef2412f"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.144351 10931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:14.145030 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.145183 10931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:14.145264 10931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "ba2f3a5913414e2397bb87d44ce3f226"
format_stamp: "Formatted at 2026-08-12 14:52:14 on dist-test-slave-6zbq"
server_key: "d863cdbf7cf342c66677e84d198c24c4"
server_key_iv: "e81957e00a0553b9511eed62cef2412f"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.145328 10931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:14.172178 10931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:14.174173 10931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:14.174305 10931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:14.174527 10931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:14.174809 10931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:14.174844 10931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.174871 10931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:14.174893 10931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.180052 10931 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:42159
I20260812 14:52:14.180092 11160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:42159 every 8 connection(s)
I20260812 14:52:14.180392 10931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:14.185372 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10931
I20260812 14:52:14.185482 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:14.185681 4204 external_mini_cluster.cc:1428] Setting key f249e79556d968ec4c5dc26733a60eee
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:14.188890 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.181081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:49765 (local address 127.4.27.62:42497)
0812 14:52:14.181155 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:14.181157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:14.181346 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:52:14.182200 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:14.182201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:14.182210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:14.182271 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:14.182839 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.183322 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.184193 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.184373 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.184416 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.186495 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:14.186496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:14.186496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:14.186502 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:14.187601 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.188040 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.188041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.188042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.188102 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.188300 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.188301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.188301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.188423 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:14.188485 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:14.188719 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 14:52:14.188807 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:52:14.189031 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.181050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42497 (local address 127.4.27.2:49765)
0812 14:52:14.181224 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:14.181264 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:14.182117 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:14.182330 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:14.182333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:14.182598 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:14.182771 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:14.182775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.183384 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.183384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:14.184134 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:14.184136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.184424 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.184424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:14.184510 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.184870 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:14.184878 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:14.186401 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:14.187673 (+ 1272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:14.187680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:14.187682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:14.187969 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:14.188158 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:14.188159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:14.188159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:14.188251 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:14.188481 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:14.188485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:14.188599 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 14:52:14.188689 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 14:52:14.188829 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260812 14:52:14.189523 11161 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42497
I20260812 14:52:14.189602 11161 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:14.189751 11161 heartbeater.cc:511] Master 127.4.27.62:42497 requested a full tablet report, sending...
I20260812 14:52:14.190089 10595 ts_manager.cc:194] Registered new tserver with Master: ba2f3a5913414e2397bb87d44ce3f226 (127.4.27.2:42159)
I20260812 14:52:14.190505 10595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:49765
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:14.208796 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42497
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:14.286690 11171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:14.286866 11171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:14.286886 11171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:14.287940 11171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:14.287971 11171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:14.287995 11171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:14.289384 11171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:14.289544 11171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:14.289707 11171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:14.294133 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:14.294251 11171 server_base.cc:1034] running on GCE node
W20260812 14:52:14.294152 11178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:14.294382 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:14.294592 11171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:14.294814 11171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:14.295953 11171 hybrid_clock.cc:648] HybridClock initialized: now 1786546334295940 us; error 27 us; skew 500 ppm
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:14.298184 11171 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:14.298592 11171 webserver.cc:460] Webserver started at http://127.4.27.3:38965/ using document root <none> and password file <none>
I20260812 14:52:14.298782 11171 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:14.298827 11171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:14.298943 11171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:14.299862 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "551ff73737474105b72b9c9419f0bc0c"
format_stamp: "Formatted at 2026-08-12 14:52:14 on dist-test-slave-6zbq"
server_key: "4b40d1e2799b064d7e7771d0470c5179"
server_key_iv: "d3319182e4632cb85b5802c8be33839f"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.300171 11171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "551ff73737474105b72b9c9419f0bc0c"
format_stamp: "Formatted at 2026-08-12 14:52:14 on dist-test-slave-6zbq"
server_key: "4b40d1e2799b064d7e7771d0470c5179"
server_key_iv: "d3319182e4632cb85b5802c8be33839f"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.301369 11171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:14.301980 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.302135 11171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:14.302213 11171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "551ff73737474105b72b9c9419f0bc0c"
format_stamp: "Formatted at 2026-08-12 14:52:14 on dist-test-slave-6zbq"
server_key: "4b40d1e2799b064d7e7771d0470c5179"
server_key_iv: "d3319182e4632cb85b5802c8be33839f"
server_key_version: "encryptionkey@0"
I20260812 14:52:14.302273 11171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:14.328871 11171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:14.330531 11171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:14.330663 11171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:14.330864 11171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:14.331144 11171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:14.331176 11171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.331204 11171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:14.331221 11171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:14.336565 11171 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:43681
I20260812 14:52:14.336606 11401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:43681 every 8 connection(s)
I20260812 14:52:14.336894 11171 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:14.342788 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11171
I20260812 14:52:14.342873 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:14.343068 4204 external_mini_cluster.cc:1428] Setting key 616afbc853b12c67545d5bfa6d267b53
I20260812 14:52:14.345444 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.337561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:52959 (local address 127.4.27.62:42497)
0812 14:52:14.337605 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:14.337607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:14.337802 (+ 195us) server_negotiation.cc:407] Connection header received
0812 14:52:14.338702 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:14.338703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:14.338711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:14.338759 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:14.339278 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.339756 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.340629 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.340863 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.340908 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.342975 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:14.342977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:14.342977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:14.342985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:14.344428 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.344763 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.344764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.344764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.344809 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.344986 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.344987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.344987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.345093 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:14.345142 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:14.345327 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 14:52:14.345385 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:52:14.345573 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.337548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42497 (local address 127.4.27.3:52959)
0812 14:52:14.337692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:14.337731 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:14.338626 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:14.338816 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:14.338820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:14.339043 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:14.339213 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:14.339217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.339815 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.339815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:14.340566 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:14.340569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.340924 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.340925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:14.341022 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.341487 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:14.341495 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:14.342894 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:14.344494 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:14.344499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:14.344501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:14.344707 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:14.344850 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:14.344851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:14.344851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:14.344934 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:14.345137 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:14.345139 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:14.345237 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 14:52:14.345310 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:52:14.345417 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260812 14:52:14.346024 11403 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42497
I20260812 14:52:14.346107 11403 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:14.346239 11403 heartbeater.cc:511] Master 127.4.27.62:42497 requested a full tablet report, sending...
I20260812 14:52:14.346560 10595 ts_manager.cc:194] Registered new tserver with Master: 551ff73737474105b72b9c9419f0bc0c (127.4.27.3:43681)
I20260812 14:52:14.346915 10595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:52959
I20260812 14:52:14.354569 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:52:14.375937 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.369202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.4.27.1:42497)
0812 14:52:14.369362 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:14.369366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:14.369375 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:14.369399 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:14.369401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:14.369413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:14.369472 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:14.369966 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.370689 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.371338 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.371620 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.371664 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.373577 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:14.373585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:14.373585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:14.373591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:14.374918 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.375219 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.375222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.375222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.375273 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.375434 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.375434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.375435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.375558 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:14.375679 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:14.375762 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:52:14.375837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[10516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:14.383360 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:14.377075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.4.27.62:42497)
0812 14:52:14.377126 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:14.377128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:14.377136 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:14.377201 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:14.377202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:14.377211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:14.377263 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:14.377693 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.378150 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.378816 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:14.378993 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:14.379043 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:14.380884 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:14.380885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:14.380885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:14.380892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:14.382162 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.382540 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.382541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.382541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.382626 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:14.382828 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:14.382829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:14.382829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:14.382980 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:14.383059 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:14.383207 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 14:52:14.383277 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 14:52:14.383592 10595 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34838
I20260812 14:52:14.384045 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10703
I20260812 14:52:14.393738 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10931
I20260812 14:52:14.410813 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11171
I20260812 14:52:14.419538 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10536
2026-08-12T14:52:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1996 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[11427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[11427](info): set up 2 sockets
Aug 12 14:52:14 dist-test-slave-6zbq krb5kdc[11427](info): commencing operation
krb5kdc: starting...
W20260812 14:52:15.593668 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:15 dist-test-slave-6zbq krb5kdc[11427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:15.704066 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:37987
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:37987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:15.786770 11447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:15.786957 11447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:15.786986 11447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:15.788085 11447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:15.788123 11447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:15.788142 11447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:15.788158 11447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:15.788184 11447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:15.789680 11447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:37987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:15.789918 11447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:15.790103 11447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:15.794512 11459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:15.794553 11455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:15.794691 11447 server_base.cc:1034] running on GCE node
W20260812 14:52:15.794572 11465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:15.794972 11447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:15.795243 11447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:15.796378 11447 hybrid_clock.cc:648] HybridClock initialized: now 1786546335796365 us; error 26 us; skew 500 ppm
Aug 12 14:52:15 dist-test-slave-6zbq krb5kdc[11427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:15.804061 11447 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:15.804363 11447 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:15.804414 11447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:15.804538 11447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:15.805536 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "1268712d6e444e1895a911d37daea163"
format_stamp: "Formatted at 2026-08-12 14:52:15 on dist-test-slave-6zbq"
server_key: "62f00aa7881cb50c5509f05ee4490c2c"
server_key_iv: "2b60d40a451895a1848a3d1d710aa4ed"
server_key_version: "encryptionkey@0"
I20260812 14:52:15.805866 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "1268712d6e444e1895a911d37daea163"
format_stamp: "Formatted at 2026-08-12 14:52:15 on dist-test-slave-6zbq"
server_key: "62f00aa7881cb50c5509f05ee4490c2c"
server_key_iv: "2b60d40a451895a1848a3d1d710aa4ed"
server_key_version: "encryptionkey@0"
I20260812 14:52:15.807070 11447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:15.807744 11473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:15.807907 11447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:15.807991 11447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "1268712d6e444e1895a911d37daea163"
format_stamp: "Formatted at 2026-08-12 14:52:15 on dist-test-slave-6zbq"
server_key: "62f00aa7881cb50c5509f05ee4490c2c"
server_key_iv: "2b60d40a451895a1848a3d1d710aa4ed"
server_key_version: "encryptionkey@0"
I20260812 14:52:15.808048 11447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:15.819260 11447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:15.820890 11447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:15.821013 11447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:15.824601 11447 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:37987
I20260812 14:52:15.824635 11572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:37987 every 8 connection(s)
I20260812 14:52:15.824870 11447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:15.825809 11573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:15.827968 11573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163: Bootstrap starting.
I20260812 14:52:15.828482 11573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:15.828720 11573 log.cc:824] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:15.829286 11573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163: No bootstrap required, opened a new log
I20260812 14:52:15.829741 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11447
I20260812 14:52:15.829836 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:15.829985 4204 external_mini_cluster.cc:1428] Setting key 48da208da2369f267f23da74ce632606
I20260812 14:52:15.831475 11573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1268712d6e444e1895a911d37daea163" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 37987 } }
I20260812 14:52:15.831643 11573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:15.831694 11573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1268712d6e444e1895a911d37daea163, State: Initialized, Role: FOLLOWER
I20260812 14:52:15.831823 11573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1268712d6e444e1895a911d37daea163" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 37987 } }
I20260812 14:52:15.831904 11573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:15.831941 11573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:15.831988 11573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:15.832546 11573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1268712d6e444e1895a911d37daea163" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 37987 } }
I20260812 14:52:15.832644 11573 leader_election.cc:302] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1268712d6e444e1895a911d37daea163; no voters:
I20260812 14:52:15.832792 11573 leader_election.cc:288] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:15.832832 11583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:15.832973 11583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [term 1 LEADER]: Becoming Leader. State: Replica: 1268712d6e444e1895a911d37daea163, State: Running, Role: LEADER
I20260812 14:52:15.833048 11573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:15.833153 11583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1268712d6e444e1895a911d37daea163" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 37987 } }
I20260812 14:52:15.833469 11583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1268712d6e444e1895a911d37daea163. Latest consensus state: current_term: 1 leader_uuid: "1268712d6e444e1895a911d37daea163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1268712d6e444e1895a911d37daea163" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 37987 } } }
I20260812 14:52:15.833521 11589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1268712d6e444e1895a911d37daea163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1268712d6e444e1895a911d37daea163" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 37987 } } }
I20260812 14:52:15.833630 11583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:15.833642 11589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:15.833858 11605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:15.834450 11605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:15.835963 11605 catalog_manager.cc:1347] Generated new cluster ID: c1e2ad4b87994a7eb3d8b7b5e4d0129c
I20260812 14:52:15.836016 11605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:52:15 dist-test-slave-6zbq krb5kdc[11427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:15.840201 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:15.830445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.4.27.62:37987)
0812 14:52:15.830591 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:15.830598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:15.830628 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:52:15.831413 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:15.831419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:15.831464 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:15.831569 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:15.832341 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:15.833447 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:15.834475 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:15.835052 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:15.835192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:15.837318 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:15.837322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:15.837323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:15.837334 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:15.838822 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:15.839127 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:15.839130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:15.839130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:15.839220 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:15.839390 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:15.839391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:15.839392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:15.839643 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:15.839785 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:15.839875 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:52:15.840000 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260812 14:52:15.853668 11605 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:15.854288 11605 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:15.857890 11605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163: Generated new TSK 0
I20260812 14:52:15.858034 11605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:15.881987 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:37987
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:15.960822 11630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:15.961001 11630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:15.961023 11630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:15.962157 11630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:15.962198 11630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:15.962225 11630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:15.963725 11630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:15.963933 11630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:15.964154 11630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:15.968725 11638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:15.968772 11641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:15.968740 11645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:15.968878 11630 server_base.cc:1034] running on GCE node
I20260812 14:52:15.969161 11630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:15.969380 11630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:15.970515 11630 hybrid_clock.cc:648] HybridClock initialized: now 1786546335970499 us; error 28 us; skew 500 ppm
Aug 12 14:52:15 dist-test-slave-6zbq krb5kdc[11427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:15.972885 11630 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:15.973189 11630 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:15.973239 11630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:15.973357 11630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:15.974311 11630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "43ff5c0164e2434b8740834e5ebac749"
format_stamp: "Formatted at 2026-08-12 14:52:15 on dist-test-slave-6zbq"
server_key: "33297795c234cd9a93cd22473e875971"
server_key_iv: "db305c05126e4002d03ebe22fd4e769c"
server_key_version: "encryptionkey@0"
I20260812 14:52:15.974661 11630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "43ff5c0164e2434b8740834e5ebac749"
format_stamp: "Formatted at 2026-08-12 14:52:15 on dist-test-slave-6zbq"
server_key: "33297795c234cd9a93cd22473e875971"
server_key_iv: "db305c05126e4002d03ebe22fd4e769c"
server_key_version: "encryptionkey@0"
I20260812 14:52:15.975795 11630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:15.976478 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:15.976626 11630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:15.976697 11630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "43ff5c0164e2434b8740834e5ebac749"
format_stamp: "Formatted at 2026-08-12 14:52:15 on dist-test-slave-6zbq"
server_key: "33297795c234cd9a93cd22473e875971"
server_key_iv: "db305c05126e4002d03ebe22fd4e769c"
server_key_version: "encryptionkey@0"
I20260812 14:52:15.976763 11630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:15.995751 11630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:15.997316 11630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:15.997442 11630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:15.997625 11630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:15.997908 11630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:15.997941 11630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:15.997967 11630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:15.997990 11630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.003201 11630 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43959
I20260812 14:52:16.003254 11868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43959 every 8 connection(s)
I20260812 14:52:16.003479 11630 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:16.006500 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11630
I20260812 14:52:16.006608 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:16.006803 4204 external_mini_cluster.cc:1428] Setting key 19035dbfe81ee7b0b9e7086d14ad735b
Aug 12 14:52:16 dist-test-slave-6zbq krb5kdc[11427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:16.012595 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.004095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:39195 (local address 127.4.27.62:37987)
0812 14:52:16.004146 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.004148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.004337 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:52:16.004751 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.004752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.004762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.004809 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.005629 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.006125 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.007456 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.007648 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.007699 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.009736 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.009737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.009737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:16.009752 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.011201 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.011693 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.011694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.011694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.011748 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.011991 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.011991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.011992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.012087 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.012134 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.012409 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.012503 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:52:16.012706 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.004074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:37987 (local address 127.4.27.1:39195)
0812 14:52:16.004227 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.004265 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.004682 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.004866 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.004869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.005101 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:16.005564 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.005569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.006228 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.006229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.007307 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.007310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.007728 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.007729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.007855 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.008275 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:16.008285 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:16.009650 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:16.011309 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:16.011317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:16.011319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:16.011627 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:16.011805 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:16.011806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:16.011806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:16.011935 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:16.012138 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:16.012143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:16.012276 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.012366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.012509 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260812 14:52:16.013101 11869 heartbeater.cc:347] Connected to a master server at 127.4.27.62:37987
I20260812 14:52:16.013206 11869 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:16.013358 11869 heartbeater.cc:511] Master 127.4.27.62:37987 requested a full tablet report, sending...
I20260812 14:52:16.013829 11504 ts_manager.cc:194] Registered new tserver with Master: 43ff5c0164e2434b8740834e5ebac749 (127.4.27.1:43959)
I20260812 14:52:16.014477 11504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:39195
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:16.031142 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:37987
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:16.109138 11875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:16.109320 11875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:16.109342 11875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:16.110415 11875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:16.110447 11875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:16.110471 11875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:16.111959 11875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:16.112120 11875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:16.112279 11875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:16.116875 11888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.116871 11890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.116873 11886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.117079 11875 server_base.cc:1034] running on GCE node
I20260812 14:52:16.117290 11875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:16.117506 11875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:16.118640 11875 hybrid_clock.cc:648] HybridClock initialized: now 1786546336118625 us; error 30 us; skew 500 ppm
Aug 12 14:52:16 dist-test-slave-6zbq krb5kdc[11427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546336, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:16.121040 11875 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:16.121464 11875 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:16.121518 11875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:16.121603 11875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:16.122614 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "90167198db284f7bb1155b77586f15cf"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "18bcf910a7fe8361e8a1110fe030c42d"
server_key_iv: "c6601a1ba1b96b76f88615faa1b9e146"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.122912 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "90167198db284f7bb1155b77586f15cf"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "18bcf910a7fe8361e8a1110fe030c42d"
server_key_iv: "c6601a1ba1b96b76f88615faa1b9e146"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.124010 11875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:16.124624 11899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.124795 11875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:16.124867 11875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "90167198db284f7bb1155b77586f15cf"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "18bcf910a7fe8361e8a1110fe030c42d"
server_key_iv: "c6601a1ba1b96b76f88615faa1b9e146"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.124917 11875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:16.160778 11875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:16.162797 11875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:16.162905 11875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:16.163072 11875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:16.163365 11875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:16.163399 11875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.163426 11875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:16.163444 11875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.168781 11875 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:37823
I20260812 14:52:16.168818 12101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:37823 every 8 connection(s)
I20260812 14:52:16.169051 11875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:52:16 dist-test-slave-6zbq krb5kdc[11427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546336, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:16.176227 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11875
I20260812 14:52:16.176316 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:16.176514 4204 external_mini_cluster.cc:1428] Setting key 3296d33a8dd4a94bc28b3b25ca1aee07
I20260812 14:52:16.177299 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.169674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:37557 (local address 127.4.27.62:37987)
0812 14:52:16.169742 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.169743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.169915 (+ 172us) server_negotiation.cc:407] Connection header received
0812 14:52:16.170382 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.170383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.170393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.170445 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.171280 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.171760 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.172660 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.172837 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.172880 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.174656 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.174657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.174658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:16.174664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.175853 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.176226 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.176228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.176228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.176295 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.176604 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.176605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.176605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.176753 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.176818 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.177094 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.177206 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:52:16.177424 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.169640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:37987 (local address 127.4.27.2:37557)
0812 14:52:16.169802 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.169841 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.170284 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.170507 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.170510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.170740 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:16.171213 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.171218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.171817 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.171818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.172602 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.172604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.172888 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.172889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.172979 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.173332 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:16.173339 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:16.174569 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:16.175910 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:16.175913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:16.175915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:16.176132 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:16.176409 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:16.176410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:16.176410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:16.176542 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:16.176820 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:16.176827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:16.176969 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.177055 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.177211 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260812 14:52:16.177932 12102 heartbeater.cc:347] Connected to a master server at 127.4.27.62:37987
I20260812 14:52:16.178012 12102 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:16.178146 12102 heartbeater.cc:511] Master 127.4.27.62:37987 requested a full tablet report, sending...
I20260812 14:52:16.178493 11504 ts_manager.cc:194] Registered new tserver with Master: 90167198db284f7bb1155b77586f15cf (127.4.27.2:37823)
I20260812 14:52:16.178856 11504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:37557
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:16.198390 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:37987
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:16.277241 12114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:16.277415 12114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:16.277437 12114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:16.278534 12114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:16.278573 12114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:16.278599 12114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:16.280056 12114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:16.280232 12114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:16.280393 12114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:16.284922 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.284940 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.284922 12122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.285022 12114 server_base.cc:1034] running on GCE node
I20260812 14:52:16.285318 12114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:16.285537 12114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:16.286674 12114 hybrid_clock.cc:648] HybridClock initialized: now 1786546336286654 us; error 29 us; skew 500 ppm
Aug 12 14:52:16 dist-test-slave-6zbq krb5kdc[11427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546336, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:16.288909 12114 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:16.289233 12114 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:16.289286 12114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:16.289412 12114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:16.290390 12114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "81fdd7c680b247a9bee1945116932701"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "d38604a540cce84af389a1806ea51b3f"
server_key_iv: "6c2fa01174a29ef69be33e18df210ce2"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.290740 12114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "81fdd7c680b247a9bee1945116932701"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "d38604a540cce84af389a1806ea51b3f"
server_key_iv: "6c2fa01174a29ef69be33e18df210ce2"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.291874 12114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:16.292546 12135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.292711 12114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:16.292795 12114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "81fdd7c680b247a9bee1945116932701"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "d38604a540cce84af389a1806ea51b3f"
server_key_iv: "6c2fa01174a29ef69be33e18df210ce2"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.292860 12114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:16.312335 12114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:16.314144 12114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:16.314273 12114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:16.314483 12114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:16.314764 12114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:16.314798 12114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.314824 12114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:16.314849 12114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.320073 12114 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:38087
I20260812 14:52:16.320107 12350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:38087 every 8 connection(s)
I20260812 14:52:16.320330 12114 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:16.322713 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12114
I20260812 14:52:16.322811 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:16.322978 4204 external_mini_cluster.cc:1428] Setting key f9ac2e8f6ae6c260d9a38baa448f3115
Aug 12 14:52:16 dist-test-slave-6zbq krb5kdc[11427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546336, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:16.328806 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.320894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:44847 (local address 127.4.27.62:37987)
0812 14:52:16.320933 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.320935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.321185 (+ 250us) server_negotiation.cc:407] Connection header received
0812 14:52:16.321586 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.321587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.321596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.321642 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.322493 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.323023 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.324191 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.324381 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.324431 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.326214 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.326215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.326215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:16.326222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.327531 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.327884 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.327885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.327886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.327945 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.328162 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.328163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.328163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.328276 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.328330 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.328627 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.328717 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:52:16.328928 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.320874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:37987 (local address 127.4.27.3:44847)
0812 14:52:16.321030 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.321069 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.321516 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.321692 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.321695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.321919 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:16.322415 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.322422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.323106 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.323107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.324110 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.324113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.324441 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.324442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.324532 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.324883 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:16.324891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:16.326136 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:16.327602 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:16.327609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:16.327611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:16.327818 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:16.328002 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:16.328003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:16.328003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:16.328108 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:16.328330 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:16.328336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:16.328486 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.328583 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.328727 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260812 14:52:16.329352 12355 heartbeater.cc:347] Connected to a master server at 127.4.27.62:37987
I20260812 14:52:16.329442 12355 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:16.329579 12355 heartbeater.cc:511] Master 127.4.27.62:37987 requested a full tablet report, sending...
I20260812 14:52:16.329909 11504 ts_manager.cc:194] Registered new tserver with Master: 81fdd7c680b247a9bee1945116932701 (127.4.27.3:38087)
I20260812 14:52:16.330268 11504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:44847
I20260812 14:52:16.334445 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:52:16.340835 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.335701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.4.27.62:37987)
0812 14:52:16.335749 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.335750 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.335837 (+ 87us) server_negotiation.cc:407] Connection header received
0812 14:52:16.335903 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.335904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.335913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.335957 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.336485 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.336947 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.337628 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.337819 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.337871 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.338871 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.338873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.338873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:16.338879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.339873 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.340242 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.340243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.340243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.340298 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:16.340444 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:16.340444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:16.340445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:16.340560 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.340646 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.340716 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.340780 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:52:16.342048 11504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:16.343564 11504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:16.349135 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.345452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.4.27.2:37823)
0812 14:52:16.345731 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.345735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.345744 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:16.345781 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.345783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.345797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.345994 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:16.346070 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.347156 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.348082 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.348761 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.348832 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.348884 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.349024 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.349061 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
I20260812 14:52:16.349125 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.345260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:37823 (local address 127.0.0.1:56860)
0812 14:52:16.345569 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.345579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.345627 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.345887 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.345888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.345889 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:16.345926 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.345928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.347272 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.347273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.348004 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.348006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.348847 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.348848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.348924 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.348938 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.349005 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.349054 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260812 14:52:16.349210 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.345423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38087 (local address 127.0.0.1:43102)
0812 14:52:16.345695 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.345704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.345770 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.345933 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.345934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.345936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:16.345989 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.345991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.346919 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.346920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.348249 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.348251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.349007 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.349008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.349061 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.349064 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.349109 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.349139 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":43,"threads_started":1}
I20260812 14:52:16.349330 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.345254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43959 (local address 127.0.0.1:38880)
0812 14:52:16.345355 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.345390 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.345451 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.345825 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.345828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.345830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:16.345892 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.345893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.347214 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.347216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.348272 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.348274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.349099 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.349100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.349172 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.349176 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.349207 (+ 31us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.349254 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":30,"thread_start_us":42,"threads_started":1}
I20260812 14:52:16.349400 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.345565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.4.27.3:38087)
0812 14:52:16.345725 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.345729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.345759 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:52:16.345855 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.345857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.345870 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.345934 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:16.346049 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.346835 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.348377 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.348859 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.348924 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.348975 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.349287 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.349328 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260812 14:52:16.349627 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.345457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.4.27.1:43959)
0812 14:52:16.345668 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.345673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.345683 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:16.345722 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.345725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.345742 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.345816 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:16.345975 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.347109 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.348330 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.349007 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.349425 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.349488 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.349513 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.349550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260812 14:52:16.349979 12234 tablet_service.cc:1467] Processing CreateTablet for tablet 5f535db532cc400b82342c5d472dade7 (DEFAULT_TABLE table=test-table [id=1faee403e8c744158059d3cb9ca4410d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:16.350164 11754 tablet_service.cc:1467] Processing CreateTablet for tablet 5f535db532cc400b82342c5d472dade7 (DEFAULT_TABLE table=test-table [id=1faee403e8c744158059d3cb9ca4410d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:16.350271 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f535db532cc400b82342c5d472dade7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:16.350517 11754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f535db532cc400b82342c5d472dade7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:16.352689 12391 tablet_bootstrap.cc:492] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Bootstrap starting.
I20260812 14:52:16.352982 12392 tablet_bootstrap.cc:492] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: Bootstrap starting.
I20260812 14:52:16.353197 11994 tablet_service.cc:1467] Processing CreateTablet for tablet 5f535db532cc400b82342c5d472dade7 (DEFAULT_TABLE table=test-table [id=1faee403e8c744158059d3cb9ca4410d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:16.353381 12391 tablet_bootstrap.cc:654] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:16.353551 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f535db532cc400b82342c5d472dade7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:16.353582 12392 tablet_bootstrap.cc:654] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:16.353935 12391 log.cc:824] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:16.353991 12392 log.cc:824] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:16.354667 12391 tablet_bootstrap.cc:492] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: No bootstrap required, opened a new log
I20260812 14:52:16.354753 12391 ts_tablet_manager.cc:1397] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:16.354959 12392 tablet_bootstrap.cc:492] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: No bootstrap required, opened a new log
I20260812 14:52:16.355019 12392 ts_tablet_manager.cc:1397] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:16.356292 12392 raft_consensus.cc:348] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.356415 12392 raft_consensus.cc:374] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:16.356434 12392 raft_consensus.cc:729] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ff5c0164e2434b8740834e5ebac749, State: Initialized, Role: FOLLOWER
I20260812 14:52:16.356523 12392 consensus_queue.cc:260] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.356665 12396 tablet_bootstrap.cc:492] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Bootstrap starting.
I20260812 14:52:16.356640 12391 raft_consensus.cc:348] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.356796 12392 ts_tablet_manager.cc:1428] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:16.356799 12391 raft_consensus.cc:374] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:16.356834 12391 raft_consensus.cc:729] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fdd7c680b247a9bee1945116932701, State: Initialized, Role: FOLLOWER
I20260812 14:52:16.356935 11869 heartbeater.cc:503] Master 127.4.27.62:37987 was elected leader, sending a full tablet report...
I20260812 14:52:16.356940 12391 consensus_queue.cc:260] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.357190 12391 ts_tablet_manager.cc:1428] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:52:16.357271 12396 tablet_bootstrap.cc:654] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:16.357322 12355 heartbeater.cc:503] Master 127.4.27.62:37987 was elected leader, sending a full tablet report...
I20260812 14:52:16.357805 12396 log.cc:824] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:16.358541 12396 tablet_bootstrap.cc:492] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: No bootstrap required, opened a new log
I20260812 14:52:16.358593 12396 ts_tablet_manager.cc:1397] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:16.359882 12396 raft_consensus.cc:348] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.359995 12396 raft_consensus.cc:374] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:16.360015 12396 raft_consensus.cc:729] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90167198db284f7bb1155b77586f15cf, State: Initialized, Role: FOLLOWER
I20260812 14:52:16.360114 12396 consensus_queue.cc:260] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.360351 12396 ts_tablet_manager.cc:1428] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:16.360359 12102 heartbeater.cc:503] Master 127.4.27.62:37987 was elected leader, sending a full tablet report...
W20260812 14:52:16.419523 12106 tablet.cc:2369] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:52:16.504086 11870 tablet.cc:2369] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:16.506197 12400 raft_consensus.cc:482] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:16.506274 12400 raft_consensus.cc:504] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.506593 12400 leader_election.cc:288] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90167198db284f7bb1155b77586f15cf (127.4.27.2:37823), 81fdd7c680b247a9bee1945116932701 (127.4.27.3:38087)
I20260812 14:52:16.509361 12390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.506965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41069 (local address 127.4.27.3:38087)
0812 14:52:16.507008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.507010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.507032 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:52:16.507084 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.507085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.507094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.507143 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:16.507285 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.507766 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.508691 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.509050 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.509118 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.509128 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.509262 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.509298 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:16.509369 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.506812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:35259 (local address 127.4.27.2:37823)
0812 14:52:16.506864 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.506867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.506878 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:52:16.506934 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.506935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.506948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.507004 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:16.507208 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.507695 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.508641 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.509024 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.509079 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.509090 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.509267 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.509306 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:52:16.509368 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.506763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:37823 (local address 127.4.27.1:35259)
0812 14:52:16.506814 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.506825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.506867 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.507027 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.507028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.507031 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:16.507135 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.507137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.507767 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.507768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.508537 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.508540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.509120 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.509121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.509198 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.509202 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.509247 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.509285 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:52:16.509379 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.506870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38087 (local address 127.4.27.1:41069)
0812 14:52:16.506977 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.506988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.507028 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.507149 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.507149 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.507151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:16.507226 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.507227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.507849 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.507850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.508608 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:16.508610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.509127 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.509128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:16.509200 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.509203 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.509249 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.509289 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260812 14:52:16.509755 12268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f535db532cc400b82342c5d472dade7" candidate_uuid: "43ff5c0164e2434b8740834e5ebac749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fdd7c680b247a9bee1945116932701" is_pre_election: true
I20260812 14:52:16.509744 12023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f535db532cc400b82342c5d472dade7" candidate_uuid: "43ff5c0164e2434b8740834e5ebac749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90167198db284f7bb1155b77586f15cf" is_pre_election: true
I20260812 14:52:16.509898 12268 raft_consensus.cc:2454] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ff5c0164e2434b8740834e5ebac749 in term 0.
I20260812 14:52:16.509898 12023 raft_consensus.cc:2454] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ff5c0164e2434b8740834e5ebac749 in term 0.
I20260812 14:52:16.510123 11662 leader_election.cc:302] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ff5c0164e2434b8740834e5ebac749, 81fdd7c680b247a9bee1945116932701; no voters:
I20260812 14:52:16.510272 12400 raft_consensus.cc:2781] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:16.510334 12400 raft_consensus.cc:482] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:16.510389 12400 raft_consensus.cc:3037] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:16.511051 12400 raft_consensus.cc:504] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.511193 12400 leader_election.cc:288] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [CANDIDATE]: Term 1 election: Requested vote from peers 90167198db284f7bb1155b77586f15cf (127.4.27.2:37823), 81fdd7c680b247a9bee1945116932701 (127.4.27.3:38087)
I20260812 14:52:16.511336 12023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f535db532cc400b82342c5d472dade7" candidate_uuid: "43ff5c0164e2434b8740834e5ebac749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90167198db284f7bb1155b77586f15cf"
I20260812 14:52:16.511379 12268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f535db532cc400b82342c5d472dade7" candidate_uuid: "43ff5c0164e2434b8740834e5ebac749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fdd7c680b247a9bee1945116932701"
I20260812 14:52:16.511408 12023 raft_consensus.cc:3037] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:16.511443 12268 raft_consensus.cc:3037] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:16.512100 12023 raft_consensus.cc:2454] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ff5c0164e2434b8740834e5ebac749 in term 1.
I20260812 14:52:16.512104 12268 raft_consensus.cc:2454] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ff5c0164e2434b8740834e5ebac749 in term 1.
I20260812 14:52:16.512279 11662 leader_election.cc:302] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ff5c0164e2434b8740834e5ebac749, 90167198db284f7bb1155b77586f15cf; no voters:
I20260812 14:52:16.512370 12400 raft_consensus.cc:2781] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:16.512527 12400 raft_consensus.cc:686] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [term 1 LEADER]: Becoming Leader. State: Replica: 43ff5c0164e2434b8740834e5ebac749, State: Running, Role: LEADER
I20260812 14:52:16.512614 12400 consensus_queue.cc:237] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } }
I20260812 14:52:16.513381 11502 catalog_manager.cc:5495] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43ff5c0164e2434b8740834e5ebac749 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "43ff5c0164e2434b8740834e5ebac749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ff5c0164e2434b8740834e5ebac749" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:16.527930 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.525760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.4.27.1:43959)
0812 14:52:16.525806 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.525808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.525815 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:16.525858 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.525859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.525876 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.525933 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:16.526074 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.526560 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.527330 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:16.527519 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:16.527566 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:16.527680 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:16.527764 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:16.527816 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.527830 (+ 14us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.527845 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:52:16.529398 12268 raft_consensus.cc:1261] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 1 FOLLOWER]: Refusing update from remote peer 43ff5c0164e2434b8740834e5ebac749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:16.529408 12023 raft_consensus.cc:1261] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 1 FOLLOWER]: Refusing update from remote peer 43ff5c0164e2434b8740834e5ebac749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:16.529673 12400 consensus_queue.cc:1035] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90167198db284f7bb1155b77586f15cf" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 37823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:16.529811 12408 consensus_queue.cc:1035] T 5f535db532cc400b82342c5d472dade7 P 43ff5c0164e2434b8740834e5ebac749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81fdd7c680b247a9bee1945116932701" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:16.533991 11502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:16.534080 11502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:16.534782 11502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1268712d6e444e1895a911d37daea163: Sending DeleteTablet for 3 replicas of tablet 5f535db532cc400b82342c5d472dade7
I20260812 14:52:16.535073 11754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f535db532cc400b82342c5d472dade7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:16 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:38880
I20260812 14:52:16.535094 12234 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f535db532cc400b82342c5d472dade7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:16 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:43102
I20260812 14:52:16.535368 12436 tablet_replica.cc:330] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: stopping tablet replica
I20260812 14:52:16.535352 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f535db532cc400b82342c5d472dade7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:16 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:56860
I20260812 14:52:16.535452 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11630
I20260812 14:52:16.535459 12436 raft_consensus.cc:2229] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:16.543268 12436 raft_consensus.cc:2258] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:16.545599 12436 ts_tablet_manager.cc:1905] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:16.547518 12436 ts_tablet_manager.cc:1918] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:16.547593 12436 log.cc:1196] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/5f535db532cc400b82342c5d472dade7
I20260812 14:52:16.547842 12436 ts_tablet_manager.cc:1939] T 5f535db532cc400b82342c5d472dade7 P 81fdd7c680b247a9bee1945116932701: Deleting consensus metadata
I20260812 14:52:16.548228 11477 catalog_manager.cc:4841] TS 81fdd7c680b247a9bee1945116932701 (127.4.27.3:38087): tablet 5f535db532cc400b82342c5d472dade7 (table test-table [id=1faee403e8c744158059d3cb9ca4410d]) successfully deleted
I20260812 14:52:16.551054 12437 tablet_replica.cc:330] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: stopping tablet replica
I20260812 14:52:16.551172 12437 raft_consensus.cc:2229] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:16.551285 12437 raft_consensus.cc:2258] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:16.551694 12437 ts_tablet_manager.cc:1905] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:16.552881 12437 ts_tablet_manager.cc:1918] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:16.552943 12437 log.cc:1196] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/5f535db532cc400b82342c5d472dade7
I20260812 14:52:16.553138 12437 ts_tablet_manager.cc:1939] T 5f535db532cc400b82342c5d472dade7 P 90167198db284f7bb1155b77586f15cf: Deleting consensus metadata
I20260812 14:52:16.553501 11477 catalog_manager.cc:4841] TS 90167198db284f7bb1155b77586f15cf (127.4.27.2:37823): tablet 5f535db532cc400b82342c5d472dade7 (table test-table [id=1faee403e8c744158059d3cb9ca4410d]) successfully deleted
W20260812 14:52:16.555344 11480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:43959 (error 108)
I20260812 14:52:16.555536 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11875
I20260812 14:52:16.557286 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.557159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43959 (local address 127.0.0.1:38902)
0812 14:52:16.557211 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.557255 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260812 14:52:16.557385 11480 catalog_manager.cc:4584] TS 43ff5c0164e2434b8740834e5ebac749 (127.4.27.1:43959): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f535db532cc400b82342c5d472dade7: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43959: connect: Connection refused (error 111)
I20260812 14:52:16.569865 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12114
I20260812 14:52:16.578047 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11447
2026-08-12T14:52:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2161 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T14:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:16Z Disabled control of system clock
I20260812 14:52:16.610049 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:35149
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:35149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:52:16.688921 12447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:16.689100 12447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:16.689121 12447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:16.690178 12447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:16.690212 12447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:16.690223 12447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:16.690234 12447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:16.690245 12447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:16.691632 12447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:35149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:35149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:16.691815 12447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:16.692039 12447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:16.696434 12456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.696532 12447 server_base.cc:1034] running on GCE node
W20260812 14:52:16.696444 12457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.696448 12459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.696920 12447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:16.697185 12447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:16.698319 12447 hybrid_clock.cc:648] HybridClock initialized: now 1786546336698309 us; error 31 us; skew 500 ppm
I20260812 14:52:16.699456 12447 webserver.cc:460] Webserver started at http://127.4.27.62:43511/ using document root <none> and password file <none>
I20260812 14:52:16.699683 12447 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:16.699743 12447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:16.699869 12447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:16.700829 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "32082507adf144d58e9c9658e3105162"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "9ad10a92d7ede5052c1f2589d7e5cdd0"
server_key_iv: "807f20375f5996901c4e1154c1833fad"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.701180 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "32082507adf144d58e9c9658e3105162"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "9ad10a92d7ede5052c1f2589d7e5cdd0"
server_key_iv: "807f20375f5996901c4e1154c1833fad"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.702450 12447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:16.703163 12468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.703308 12447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:16.703392 12447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "32082507adf144d58e9c9658e3105162"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "9ad10a92d7ede5052c1f2589d7e5cdd0"
server_key_iv: "807f20375f5996901c4e1154c1833fad"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.703459 12447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:16.706552 12447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:16.706789 12447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:16.706902 12447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:16.710532 12447 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:35149
I20260812 14:52:16.710572 12561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:35149 every 8 connection(s)
I20260812 14:52:16.710855 12447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:16.711431 12563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:16.713748 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12447
I20260812 14:52:16.713841 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:16.714020 4204 external_mini_cluster.cc:1428] Setting key b0fb20b8fdc7cf2f06350fa3fdcfe7fa
I20260812 14:52:16.714213 12563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162: Bootstrap starting.
I20260812 14:52:16.714953 12563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:16.715230 12563 log.cc:824] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:16.715898 12563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162: No bootstrap required, opened a new log
I20260812 14:52:16.716678 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.714556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.4.27.62:35149)
0812 14:52:16.714726 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.714733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.714756 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:52:16.715184 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.715191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.715239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.715362 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.716242 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.716247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.716247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:16.716252 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.716302 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:16.716315 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.716409 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.716440 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.716494 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260812 14:52:16.717518 12563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32082507adf144d58e9c9658e3105162" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35149 } }
I20260812 14:52:16.717653 12563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:16.717689 12563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32082507adf144d58e9c9658e3105162, State: Initialized, Role: FOLLOWER
I20260812 14:52:16.717805 12563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32082507adf144d58e9c9658e3105162" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35149 } }
I20260812 14:52:16.718197 12563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:16.718245 12563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:16.718294 12563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:16.718807 12563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32082507adf144d58e9c9658e3105162" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35149 } }
I20260812 14:52:16.718899 12563 leader_election.cc:302] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32082507adf144d58e9c9658e3105162; no voters:
I20260812 14:52:16.719039 12563 leader_election.cc:288] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:16.719076 12574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:16.719244 12574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [term 1 LEADER]: Becoming Leader. State: Replica: 32082507adf144d58e9c9658e3105162, State: Running, Role: LEADER
I20260812 14:52:16.719285 12563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:16.719347 12574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32082507adf144d58e9c9658e3105162" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35149 } }
I20260812 14:52:16.719726 12578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32082507adf144d58e9c9658e3105162. Latest consensus state: current_term: 1 leader_uuid: "32082507adf144d58e9c9658e3105162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32082507adf144d58e9c9658e3105162" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35149 } } }
I20260812 14:52:16.719756 12575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32082507adf144d58e9c9658e3105162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32082507adf144d58e9c9658e3105162" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35149 } } }
I20260812 14:52:16.719826 12578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:16.719832 12575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:16.720021 12592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:16.720460 12592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:16.722219 12592 catalog_manager.cc:1347] Generated new cluster ID: 135e93b788c6434d83d59ffc2413e2c2
I20260812 14:52:16.722278 12592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:16.738636 12592 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:16.738766 12592 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:16.749534 12592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162: Generated new TSK 0
I20260812 14:52:16.749686 12592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:16.760326 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:35149
--builtin_ntp_servers=127.4.27.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 14:52:16.840407 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:16.840593 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:16.840615 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:16.841758 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:16.841794 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:16.841820 12608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:16.843168 12608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:35149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:16.843384 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:16.843648 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:16.848287 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.848276 12619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.848304 12608 server_base.cc:1034] running on GCE node
W20260812 14:52:16.848278 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.848732 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:16.848948 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:16.850085 12608 hybrid_clock.cc:648] HybridClock initialized: now 1786546336850070 us; error 31 us; skew 500 ppm
I20260812 14:52:16.851303 12608 webserver.cc:460] Webserver started at http://127.4.27.1:44231/ using document root <none> and password file <none>
I20260812 14:52:16.851517 12608 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:16.851562 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:16.851686 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:16.852651 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "875787165e8c47c7b324e0c9e7260164"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "e33596dffcabde034f0ce1708c9caa76"
server_key_iv: "a9dcbc8fb6845f63d7a7958ecca9abcd"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.852957 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "875787165e8c47c7b324e0c9e7260164"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "e33596dffcabde034f0ce1708c9caa76"
server_key_iv: "a9dcbc8fb6845f63d7a7958ecca9abcd"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.854138 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:16.854833 12629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.854977 12608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.855054 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "875787165e8c47c7b324e0c9e7260164"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "e33596dffcabde034f0ce1708c9caa76"
server_key_iv: "a9dcbc8fb6845f63d7a7958ecca9abcd"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.855136 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:16.858150 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:16.858404 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:16.858517 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:16.858703 12608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:16.859021 12608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:16.859059 12608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.859095 12608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:16.859120 12608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.864492 12608 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43863
I20260812 14:52:16.864549 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43863 every 8 connection(s)
I20260812 14:52:16.864815 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:16.865535 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12608
I20260812 14:52:16.865646 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:16.865803 4204 external_mini_cluster.cc:1428] Setting key c91fbcf5d681f4296526cb5aa6b6805c
I20260812 14:52:16.866688 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:35149
--builtin_ntp_servers=127.4.27.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 14:52:16.867654 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.865435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:56775 (local address 127.4.27.62:35149)
0812 14:52:16.865501 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.865503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.865712 (+ 209us) server_negotiation.cc:407] Connection header received
0812 14:52:16.866225 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.866226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.866232 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.866282 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.867430 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.867432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.867432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:16.867437 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.867452 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:16.867456 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.867505 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.867574 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.867587 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:52:16.867837 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.865425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:35149 (local address 127.4.27.1:56775)
0812 14:52:16.865593 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.865629 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.866136 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.866377 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.866381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.866391 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:16.867296 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:16.867297 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:16.867324 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:16.867351 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:16.867510 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:16.867516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:16.867519 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.867586 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.867644 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260812 14:52:16.868248 12830 heartbeater.cc:347] Connected to a master server at 127.4.27.62:35149
I20260812 14:52:16.868333 12830 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:16.868469 12830 heartbeater.cc:511] Master 127.4.27.62:35149 requested a full tablet report, sending...
I20260812 14:52:16.868945 12498 ts_manager.cc:194] Registered new tserver with Master: 875787165e8c47c7b324e0c9e7260164 (127.4.27.1:43863)
W20260812 14:52:16.946255 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:16.946472 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:16.946496 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:16.947587 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:16.947616 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:16.947639 12835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:16.948871 12835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:35149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:16.949033 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:16.949254 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:16.953727 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:16.953753 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.953876 12835 server_base.cc:1034] running on GCE node
W20260812 14:52:16.953753 12845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:16.954090 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:16.954298 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:16.955449 12835 hybrid_clock.cc:648] HybridClock initialized: now 1786546336955432 us; error 33 us; skew 500 ppm
I20260812 14:52:16.956539 12835 webserver.cc:460] Webserver started at http://127.4.27.2:44893/ using document root <none> and password file <none>
I20260812 14:52:16.956743 12835 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:16.956790 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:16.956916 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:16.957950 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "d434c2247b5349e4b045e51765e6fec9"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "3c71e40698c582cd2d6d663964ae3909"
server_key_iv: "313b6e80ccf8cd661c27ec6b10dcbe19"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.958272 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "d434c2247b5349e4b045e51765e6fec9"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "3c71e40698c582cd2d6d663964ae3909"
server_key_iv: "313b6e80ccf8cd661c27ec6b10dcbe19"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.959477 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:16.960223 12864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.960438 12835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:16.960536 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "d434c2247b5349e4b045e51765e6fec9"
format_stamp: "Formatted at 2026-08-12 14:52:16 on dist-test-slave-6zbq"
server_key: "3c71e40698c582cd2d6d663964ae3909"
server_key_iv: "313b6e80ccf8cd661c27ec6b10dcbe19"
server_key_version: "encryptionkey@0"
I20260812 14:52:16.960604 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:16.963984 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:16.964221 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:16.964339 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:16.964527 12835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:16.964808 12835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:16.964840 12835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.964865 12835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:16.964890 12835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:16.970151 12835 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:40815
I20260812 14:52:16.970187 13064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:40815 every 8 connection(s)
I20260812 14:52:16.970502 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:16.971750 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12835
I20260812 14:52:16.971850 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:16.972047 4204 external_mini_cluster.cc:1428] Setting key 165bce2cb2efa8e707474c134e841323
I20260812 14:52:16.973217 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:35149
--builtin_ntp_servers=127.4.27.20:37419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 14:52:16.973465 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.971116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:38233 (local address 127.4.27.62:35149)
0812 14:52:16.971170 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:16.971181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:16.971368 (+ 187us) server_negotiation.cc:407] Connection header received
0812 14:52:16.971922 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:16.971925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:16.971933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:16.971964 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:16.973202 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:16.973203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:16.973204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:16.973208 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:16.973223 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:16.973227 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:16.973273 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:16.973384 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 14:52:16.973400 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:52:16.973819 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:16.971105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:35149 (local address 127.4.27.2:38233)
0812 14:52:16.971265 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:16.971295 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:16.971792 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:16.972145 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:16.972149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:16.972158 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:16.973048 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:16.973049 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:16.973084 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:16.973116 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:16.973284 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:16.973290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:16.973293 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:16.973561 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 14:52:16.973621 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260812 14:52:16.974291 13065 heartbeater.cc:347] Connected to a master server at 127.4.27.62:35149
I20260812 14:52:16.974404 13065 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:16.974545 13065 heartbeater.cc:511] Master 127.4.27.62:35149 requested a full tablet report, sending...
I20260812 14:52:16.974864 12498 ts_manager.cc:194] Registered new tserver with Master: d434c2247b5349e4b045e51765e6fec9 (127.4.27.2:40815)
W20260812 14:52:17.052632 13068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:17.052820 13068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:17.052841 13068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:17.053903 13068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:17.053933 13068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:17.053956 13068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:17.055320 13068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:35149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:17.055487 13068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:17.055706 13068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:17.060187 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.060300 13068 server_base.cc:1034] running on GCE node
W20260812 14:52:17.060197 13078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:17.060197 13081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.060568 13068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:17.060801 13068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:17.061933 13068 hybrid_clock.cc:648] HybridClock initialized: now 1786546337061921 us; error 29 us; skew 500 ppm
I20260812 14:52:17.063153 13068 webserver.cc:460] Webserver started at http://127.4.27.3:35127/ using document root <none> and password file <none>
I20260812 14:52:17.063355 13068 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:17.063397 13068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:17.063517 13068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:17.064509 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "bb2f48732ea246c79dce232eb9a59d62"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "d728204438b629dbdefc7b27ca846150"
server_key_iv: "93b16ad69f7b088ae7c09ff5769d9568"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.064823 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "bb2f48732ea246c79dce232eb9a59d62"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "d728204438b629dbdefc7b27ca846150"
server_key_iv: "93b16ad69f7b088ae7c09ff5769d9568"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.065953 13068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:17.066624 13097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:17.066756 13068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:17.066838 13068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "bb2f48732ea246c79dce232eb9a59d62"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "d728204438b629dbdefc7b27ca846150"
server_key_iv: "93b16ad69f7b088ae7c09ff5769d9568"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.066896 13068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:17.069968 13068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:17.070185 13068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:17.070300 13068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:17.070508 13068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:17.070777 13068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:17.070808 13068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:17.070833 13068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:17.070856 13068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:17.076048 13068 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:39721
I20260812 14:52:17.076087 13308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:39721 every 8 connection(s)
I20260812 14:52:17.076350 13068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:17.078177 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13068
I20260812 14:52:17.078266 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:17.078459 4204 external_mini_cluster.cc:1428] Setting key fd020a6e129c03f1f4d6510de0ae4b7a
I20260812 14:52:17.078996 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.076965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:56455 (local address 127.4.27.62:35149)
0812 14:52:17.077021 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.077023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.077212 (+ 189us) server_negotiation.cc:407] Connection header received
0812 14:52:17.077578 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.077578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.077591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.077641 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.078754 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.078756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.078757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.078762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.078780 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.078784 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.078838 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.078911 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.078928 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:52:17.079161 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.076945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:35149 (local address 127.4.27.3:56455)
0812 14:52:17.077108 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.077143 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.077511 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.077693 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.077696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.077703 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:17.078614 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:17.078615 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:17.078641 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:17.078672 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:17.078843 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:17.078849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:17.078852 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:17.078919 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:52:17.078975 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260812 14:52:17.079607 13309 heartbeater.cc:347] Connected to a master server at 127.4.27.62:35149
I20260812 14:52:17.079698 13309 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:17.079829 13309 heartbeater.cc:511] Master 127.4.27.62:35149 requested a full tablet report, sending...
I20260812 14:52:17.080123 12498 ts_manager.cc:194] Registered new tserver with Master: bb2f48732ea246c79dce232eb9a59d62 (127.4.27.3:39721)
I20260812 14:52:17.089839 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:52:17.091797 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.091026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.4.27.62:35149)
0812 14:52:17.091073 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.091074 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.091152 (+ 78us) server_negotiation.cc:407] Connection header received
0812 14:52:17.091202 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.091203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.091206 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.091251 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.091660 (+ 409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.091661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.091661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.091665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.091676 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.091679 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.091717 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.091747 (+ 30us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.091757 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:52:17.093305 12498 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:17.094826 12498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:17.098133 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.096689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:39721 (local address 127.0.0.1:55156)
0812 14:52:17.096945 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.096954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.096986 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.097165 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.097170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.097175 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:17.097717 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:17.097718 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:17.097733 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:17.097754 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:17.097988 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:17.097994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:17.097997 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:17.098060 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:52:17.098072 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260812 14:52:17.098165 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.096918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.4.27.3:39721)
0812 14:52:17.097049 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.097053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.097063 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:17.097084 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.097086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.097094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.097154 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.097836 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.097868 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.097868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.097873 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.097908 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.097926 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.097976 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.098074 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.098093 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260812 14:52:17.098398 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.096689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:40815 (local address 127.0.0.1:52498)
0812 14:52:17.096799 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.096837 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.096889 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.097651 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.097652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.097654 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:17.097996 (+ 342us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:17.097997 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:17.098005 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:17.098017 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:17.098274 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:17.098275 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:17.098276 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:52:17.098304 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 14:52:17.098317 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
I20260812 14:52:17.098665 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.096913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.4.27.2:40815)
0812 14:52:17.097496 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.097501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.097510 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:17.097534 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.097536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.097544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.097648 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.098102 (+ 454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.098112 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.098112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.098117 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.098160 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.098193 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.098548 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.098573 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.098588 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":73,"threads_started":1}
I20260812 14:52:17.098842 13189 tablet_service.cc:1467] Processing CreateTablet for tablet c9894de4ee2f4ce581bbdb506b7c5e85 (DEFAULT_TABLE table=test-table [id=83f7cda151764d38ac151471b440c0f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:17.099159 13189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9894de4ee2f4ce581bbdb506b7c5e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:17.099216 12946 tablet_service.cc:1467] Processing CreateTablet for tablet c9894de4ee2f4ce581bbdb506b7c5e85 (DEFAULT_TABLE table=test-table [id=83f7cda151764d38ac151471b440c0f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:17.099561 12946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9894de4ee2f4ce581bbdb506b7c5e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:17.101512 13337 tablet_bootstrap.cc:492] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Bootstrap starting.
I20260812 14:52:17.101727 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.096701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43863 (local address 127.0.0.1:34756)
0812 14:52:17.097054 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.097061 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.097088 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.097575 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.097577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.097579 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:17.098291 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:17.098292 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:17.098299 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:17.098309 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:17.101602 (+ 3293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:17.101603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:17.101603 (+ 0us) client_negotiation.cc:769] Sending connection context
0812 14:52:17.101643 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:52:17.101666 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":104,"threads_started":1}
I20260812 14:52:17.101912 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.097061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.4.27.1:43863)
0812 14:52:17.097219 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.097223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.097233 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:17.097257 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.097258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.097265 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.097335 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.098409 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.098419 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.098419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.098424 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.101490 (+ 3066us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.101522 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.101578 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.101821 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.101839 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260812 14:52:17.102214 13337 tablet_bootstrap.cc:654] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:17.102398 13338 tablet_bootstrap.cc:492] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Bootstrap starting.
I20260812 14:52:17.102550 13337 log.cc:824] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:17.102612 12719 tablet_service.cc:1467] Processing CreateTablet for tablet c9894de4ee2f4ce581bbdb506b7c5e85 (DEFAULT_TABLE table=test-table [id=83f7cda151764d38ac151471b440c0f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:17.102944 12719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9894de4ee2f4ce581bbdb506b7c5e85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:17.102988 13338 tablet_bootstrap.cc:654] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:17.103286 13338 log.cc:824] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:17.103403 13337 tablet_bootstrap.cc:492] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: No bootstrap required, opened a new log
I20260812 14:52:17.103467 13337 ts_tablet_manager.cc:1397] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:17.104038 13338 tablet_bootstrap.cc:492] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: No bootstrap required, opened a new log
I20260812 14:52:17.104104 13338 ts_tablet_manager.cc:1397] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:52:17.105450 13341 tablet_bootstrap.cc:492] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: Bootstrap starting.
I20260812 14:52:17.105830 13337 raft_consensus.cc:348] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.106010 13337 raft_consensus.cc:374] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:17.106045 13337 raft_consensus.cc:729] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2f48732ea246c79dce232eb9a59d62, State: Initialized, Role: FOLLOWER
I20260812 14:52:17.106158 13341 tablet_bootstrap.cc:654] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:17.106185 13337 consensus_queue.cc:260] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.106483 13309 heartbeater.cc:503] Master 127.4.27.62:35149 was elected leader, sending a full tablet report...
I20260812 14:52:17.106545 13341 log.cc:824] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:17.106524 13338 raft_consensus.cc:348] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.106662 13338 raft_consensus.cc:374] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:17.106698 13338 raft_consensus.cc:729] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d434c2247b5349e4b045e51765e6fec9, State: Initialized, Role: FOLLOWER
I20260812 14:52:17.106812 13338 consensus_queue.cc:260] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.107056 13338 ts_tablet_manager.cc:1428] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:52:17.107131 13337 ts_tablet_manager.cc:1428] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 14:52:17.107170 13065 heartbeater.cc:503] Master 127.4.27.62:35149 was elected leader, sending a full tablet report...
I20260812 14:52:17.107389 13341 tablet_bootstrap.cc:492] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: No bootstrap required, opened a new log
I20260812 14:52:17.107504 13341 ts_tablet_manager.cc:1397] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:17.109376 13341 raft_consensus.cc:348] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.109493 13341 raft_consensus.cc:374] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:17.109524 13341 raft_consensus.cc:729] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875787165e8c47c7b324e0c9e7260164, State: Initialized, Role: FOLLOWER
I20260812 14:52:17.109627 13341 consensus_queue.cc:260] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.109836 13341 ts_tablet_manager.cc:1428] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:17.109894 12830 heartbeater.cc:503] Master 127.4.27.62:35149 was elected leader, sending a full tablet report...
W20260812 14:52:17.115281 12831 tablet.cc:2369] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:17.172291 13347 raft_consensus.cc:482] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:17.172421 13347 raft_consensus.cc:504] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.172735 13347 leader_election.cc:288] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb2f48732ea246c79dce232eb9a59d62 (127.4.27.3:39721), 875787165e8c47c7b324e0c9e7260164 (127.4.27.1:43863)
I20260812 14:52:17.173889 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.172891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:39721 (local address 127.4.27.2:55989)
0812 14:52:17.172957 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.172969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.173022 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.173153 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.173154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.173156 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:17.173609 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:17.173610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:17.173617 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:17.173629 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:17.173777 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:17.173777 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:17.173778 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:52:17.173822 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:52:17.173835 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:52:17.173905 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.172934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:55989 (local address 127.4.27.3:39721)
0812 14:52:17.172981 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.172982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.173018 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:52:17.173074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.173075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.173081 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.173140 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.173687 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.173689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.173689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.173693 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.173713 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.173719 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.173773 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.173824 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.173838 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:17.174063 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.172891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43863 (local address 127.4.27.2:46325)
0812 14:52:17.173115 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.173124 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.173187 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.173310 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.173311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.173313 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:17.173751 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:17.173751 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:17.173756 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:17.173767 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:17.173928 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:17.173928 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:17.173929 (+ 1us) client_negotiation.cc:769] Sending connection context
0812 14:52:17.173976 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:52:17.173989 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":55,"thread_start_us":103,"threads_started":1}
I20260812 14:52:17.174086 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.172957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:46325 (local address 127.4.27.1:43863)
0812 14:52:17.172997 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.172999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.173187 (+ 188us) server_negotiation.cc:407] Connection header received
0812 14:52:17.173236 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.173237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.173243 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.173300 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.173847 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.173849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.173849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.173853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.173868 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.173873 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.173924 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.174013 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.174028 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:17.174258 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9894de4ee2f4ce581bbdb506b7c5e85" candidate_uuid: "d434c2247b5349e4b045e51765e6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2f48732ea246c79dce232eb9a59d62" is_pre_election: true
I20260812 14:52:17.174415 13220 raft_consensus.cc:2454] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d434c2247b5349e4b045e51765e6fec9 in term 0.
I20260812 14:52:17.174433 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9894de4ee2f4ce581bbdb506b7c5e85" candidate_uuid: "d434c2247b5349e4b045e51765e6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875787165e8c47c7b324e0c9e7260164" is_pre_election: true
I20260812 14:52:17.174556 12746 raft_consensus.cc:2454] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d434c2247b5349e4b045e51765e6fec9 in term 0.
I20260812 14:52:17.174652 12872 leader_election.cc:302] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb2f48732ea246c79dce232eb9a59d62, d434c2247b5349e4b045e51765e6fec9; no voters:
I20260812 14:52:17.174765 13347 raft_consensus.cc:2781] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:17.174829 13347 raft_consensus.cc:482] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:17.174844 13347 raft_consensus.cc:3037] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:17.175489 13347 raft_consensus.cc:504] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.175594 13347 leader_election.cc:288] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [CANDIDATE]: Term 1 election: Requested vote from peers bb2f48732ea246c79dce232eb9a59d62 (127.4.27.3:39721), 875787165e8c47c7b324e0c9e7260164 (127.4.27.1:43863)
I20260812 14:52:17.175720 13220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9894de4ee2f4ce581bbdb506b7c5e85" candidate_uuid: "d434c2247b5349e4b045e51765e6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2f48732ea246c79dce232eb9a59d62"
I20260812 14:52:17.175755 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9894de4ee2f4ce581bbdb506b7c5e85" candidate_uuid: "d434c2247b5349e4b045e51765e6fec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875787165e8c47c7b324e0c9e7260164"
I20260812 14:52:17.175796 13220 raft_consensus.cc:3037] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:17.175827 12746 raft_consensus.cc:3037] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:17.176472 12746 raft_consensus.cc:2454] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d434c2247b5349e4b045e51765e6fec9 in term 1.
I20260812 14:52:17.176493 13220 raft_consensus.cc:2454] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d434c2247b5349e4b045e51765e6fec9 in term 1.
I20260812 14:52:17.176635 12871 leader_election.cc:302] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 875787165e8c47c7b324e0c9e7260164, d434c2247b5349e4b045e51765e6fec9; no voters:
I20260812 14:52:17.176713 13347 raft_consensus.cc:2781] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:17.176843 13347 raft_consensus.cc:686] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 1 LEADER]: Becoming Leader. State: Replica: d434c2247b5349e4b045e51765e6fec9, State: Running, Role: LEADER
I20260812 14:52:17.176911 13347 consensus_queue.cc:237] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } }
I20260812 14:52:17.177636 12498 catalog_manager.cc:5495] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d434c2247b5349e4b045e51765e6fec9 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "d434c2247b5349e4b045e51765e6fec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d434c2247b5349e4b045e51765e6fec9" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 40815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:17.193063 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.192298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.4.27.2:40815)
0812 14:52:17.192358 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.192359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.192367 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:17.192394 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.192394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.192404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.192448 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.192912 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.192913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.192913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.192915 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.192925 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.192928 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.192959 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.192995 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.193004 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:52:17.194540 13220 raft_consensus.cc:1261] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 1 FOLLOWER]: Refusing update from remote peer d434c2247b5349e4b045e51765e6fec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:17.194569 12746 raft_consensus.cc:1261] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 1 FOLLOWER]: Refusing update from remote peer d434c2247b5349e4b045e51765e6fec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:17.194833 13347 consensus_queue.cc:1035] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f48732ea246c79dce232eb9a59d62" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:17.194967 13350 consensus_queue.cc:1035] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875787165e8c47c7b324e0c9e7260164" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:17.197430 13366 mvcc.cc:205] Tried to move back new op lower bound from 7317693797146632192 to 7317693797077192704. Current Snapshot: MvccSnapshot[applied={T|T < 7317693797146632192}]
I20260812 14:52:17.197943 13365 mvcc.cc:205] Tried to move back new op lower bound from 7317693797146632192 to 7317693797077192704. Current Snapshot: MvccSnapshot[applied={T|T < 7317693797146632192}]
I20260812 14:52:17.199301 12496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:17.199393 12496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:17.200069 12496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32082507adf144d58e9c9658e3105162: Sending DeleteTablet for 3 replicas of tablet c9894de4ee2f4ce581bbdb506b7c5e85
I20260812 14:52:17.200371 12719 tablet_service.cc:1514] Processing DeleteTablet for tablet c9894de4ee2f4ce581bbdb506b7c5e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:17 UTC) from {username='slave'} at 127.0.0.1:34756
I20260812 14:52:17.200415 12946 tablet_service.cc:1514] Processing DeleteTablet for tablet c9894de4ee2f4ce581bbdb506b7c5e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:17 UTC) from {username='slave'} at 127.0.0.1:52498
I20260812 14:52:17.200598 13376 tablet_replica.cc:330] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164: stopping tablet replica
I20260812 14:52:17.200605 13377 tablet_replica.cc:330] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: stopping tablet replica
I20260812 14:52:17.200588 13189 tablet_service.cc:1514] Processing DeleteTablet for tablet c9894de4ee2f4ce581bbdb506b7c5e85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:17 UTC) from {username='slave'} at 127.0.0.1:55156
I20260812 14:52:17.200690 13377 raft_consensus.cc:2229] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:17.200691 13376 raft_consensus.cc:2229] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:17.200778 13377 raft_consensus.cc:2258] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:17.200786 13376 raft_consensus.cc:2258] T c9894de4ee2f4ce581bbdb506b7c5e85 P 875787165e8c47c7b324e0c9e7260164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:17.200878 13379 tablet_replica.cc:330] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: stopping tablet replica
I20260812 14:52:17.200987 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12608
I20260812 14:52:17.200994 13379 raft_consensus.cc:2229] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:17.201512 13379 raft_consensus.cc:2258] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:17.201678 13377 ts_tablet_manager.cc:1905] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:17.208940 13379 ts_tablet_manager.cc:1905] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:17.209754 13377 ts_tablet_manager.cc:1918] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:17.209829 13377 log.cc:1196] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/c9894de4ee2f4ce581bbdb506b7c5e85
I20260812 14:52:17.210084 13377 ts_tablet_manager.cc:1939] T c9894de4ee2f4ce581bbdb506b7c5e85 P d434c2247b5349e4b045e51765e6fec9: Deleting consensus metadata
I20260812 14:52:17.210300 13379 ts_tablet_manager.cc:1918] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:17.210387 13379 log.cc:1196] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/c9894de4ee2f4ce581bbdb506b7c5e85
I20260812 14:52:17.210552 12474 catalog_manager.cc:4841] TS d434c2247b5349e4b045e51765e6fec9 (127.4.27.2:40815): tablet c9894de4ee2f4ce581bbdb506b7c5e85 (table test-table [id=83f7cda151764d38ac151471b440c0f5]) successfully deleted
I20260812 14:52:17.210636 13379 ts_tablet_manager.cc:1939] T c9894de4ee2f4ce581bbdb506b7c5e85 P bb2f48732ea246c79dce232eb9a59d62: Deleting consensus metadata
I20260812 14:52:17.211292 12477 catalog_manager.cc:4841] TS bb2f48732ea246c79dce232eb9a59d62 (127.4.27.3:39721): tablet c9894de4ee2f4ce581bbdb506b7c5e85 (table test-table [id=83f7cda151764d38ac151471b440c0f5]) successfully deleted
W20260812 14:52:17.236517 12475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:43863 (error 108)
I20260812 14:52:17.236850 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12835
I20260812 14:52:17.237498 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.236919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43863 (local address 127.0.0.1:34762)
0812 14:52:17.237413 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.237464 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469}
W20260812 14:52:17.238818 12475 catalog_manager.cc:4584] TS 875787165e8c47c7b324e0c9e7260164 (127.4.27.1:43863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9894de4ee2f4ce581bbdb506b7c5e85: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43863: connect: Connection refused (error 111)
I20260812 14:52:17.245659 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13068
I20260812 14:52:17.258248 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12447
2026-08-12T14:52:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (681 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 14:52:17.283578 4204 mini_oidc.cc:152] Starting JWKS server
I20260812 14:52:17.284767 4204 webserver.cc:460] Webserver started at https://127.0.0.1:41627/ using document root <none> and password file <none>
I20260812 14:52:17.284857 4204 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:52:17.285017 4204 webserver.cc:460] Webserver started at http://127.0.0.1:33541/ using document root <none> and password file <none>
2026-08-12T14:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:17Z Disabled control of system clock
I20260812 14:52:17.292332 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39065
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:39373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39065
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41627/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/testchainca.cert with env {}
W20260812 14:52:17.369489 13394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:17.369678 13394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:17.369699 13394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:17.370843 13394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:17.370882 13394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:17.370893 13394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:17.370905 13394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:17.370929 13394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:52:17.370952 13394 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:52:17.370971 13394 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41627/jwks.json
I20260812 14:52:17.372455 13394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41627/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:17.372678 13394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:17.372866 13394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:17.377245 13401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:17.377357 13406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:17.377259 13402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.377322 13394 server_base.cc:1034] running on GCE node
I20260812 14:52:17.377642 13394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:17.377889 13394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:17.379010 13394 hybrid_clock.cc:648] HybridClock initialized: now 1786546337379000 us; error 26 us; skew 500 ppm
I20260812 14:52:17.380062 13394 webserver.cc:460] Webserver started at http://127.4.27.62:34331/ using document root <none> and password file <none>
I20260812 14:52:17.380292 13394 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:17.380337 13394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:17.380453 13394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:17.381388 13394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "b4036b9f1d29483ebd967812061b77ef"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "06abd5e5331d1d2a1f7304de631a637d"
server_key_iv: "7b8918f558c500415ac4ac04fbbeacbc"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.381697 13394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "b4036b9f1d29483ebd967812061b77ef"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "06abd5e5331d1d2a1f7304de631a637d"
server_key_iv: "7b8918f558c500415ac4ac04fbbeacbc"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.382885 13394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:17.383502 13413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:17.383658 13394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:17.383746 13394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "b4036b9f1d29483ebd967812061b77ef"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "06abd5e5331d1d2a1f7304de631a637d"
server_key_iv: "7b8918f558c500415ac4ac04fbbeacbc"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.383798 13394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:17.408717 13394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:17.408929 13394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:17.409019 13394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:17.412591 13394 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:39065
I20260812 14:52:17.412626 13525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:39065 every 8 connection(s)
I20260812 14:52:17.412914 13394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:17.413794 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:17.415961 13526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef: Bootstrap starting.
I20260812 14:52:17.416059 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13394
I20260812 14:52:17.416148 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:17.416332 4204 external_mini_cluster.cc:1428] Setting key 2c81ffcf1937370035592ef449304957
I20260812 14:52:17.416603 13526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:17.417128 13526 log.cc:824] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:17.417862 13526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef: No bootstrap required, opened a new log
I20260812 14:52:17.420013 13526 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4036b9f1d29483ebd967812061b77ef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39065 } }
I20260812 14:52:17.420171 13526 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:17.420207 13526 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4036b9f1d29483ebd967812061b77ef, State: Initialized, Role: FOLLOWER
I20260812 14:52:17.420325 13526 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4036b9f1d29483ebd967812061b77ef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39065 } }
I20260812 14:52:17.420395 13526 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:17.420429 13526 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:17.420471 13526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:17.421120 13526 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4036b9f1d29483ebd967812061b77ef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39065 } }
I20260812 14:52:17.421236 13526 leader_election.cc:302] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4036b9f1d29483ebd967812061b77ef; no voters:
I20260812 14:52:17.421391 13526 leader_election.cc:288] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:17.421432 13532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:17.421545 13532 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [term 1 LEADER]: Becoming Leader. State: Replica: b4036b9f1d29483ebd967812061b77ef, State: Running, Role: LEADER
I20260812 14:52:17.421627 13532 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4036b9f1d29483ebd967812061b77ef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39065 } }
I20260812 14:52:17.421654 13526 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:17.422061 13532 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4036b9f1d29483ebd967812061b77ef. Latest consensus state: current_term: 1 leader_uuid: "b4036b9f1d29483ebd967812061b77ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4036b9f1d29483ebd967812061b77ef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39065 } } }
I20260812 14:52:17.422170 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.416774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.4.27.62:39065)
0812 14:52:17.416953 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.416961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.416996 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:52:17.417470 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.417477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.417537 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.417679 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.417879 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.419236 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.419991 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.420572 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.420696 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.421270 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.421274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.421275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.421469 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.421511 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.421524 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.421801 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.421832 (+ 31us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.421908 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260812 14:52:17.422133 13534 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4036b9f1d29483ebd967812061b77ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4036b9f1d29483ebd967812061b77ef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39065 } } }
I20260812 14:52:17.422227 13534 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:17.422433 13541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:17.422904 13541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:17.422137 13532 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:17.423014 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:52:17.424755 13541 catalog_manager.cc:1347] Generated new cluster ID: 73a8a3b229574d34a906dd618b4c71e8
I20260812 14:52:17.424811 13541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:17.431028 13541 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:17.431653 13541 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:17.439952 13541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4036b9f1d29483ebd967812061b77ef: Generated new TSK 0
I20260812 14:52:17.440099 13541 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 14:52:17.460812 13573 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 14:52:17.460938 13573 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 14:52:17.461117 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.446658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.4.27.62:39065)
0812 14:52:17.446704 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.446706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.446837 (+ 131us) server_negotiation.cc:407] Connection header received
0812 14:52:17.446915 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.446915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.446926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.446975 (+ 49us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:52:17.447111 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.447577 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.448293 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.448495 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.448547 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.448727 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:52:17.460870 (+ 12143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 14:52:17.461051 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260812 14:52:17.461148 13529 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 14:52:17.461560 13572 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:17.446596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39065 (local address 127.0.0.1:34234)
0812 14:52:17.446781 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.446788 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.446843 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.446991 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.446992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.446994 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:52:17.447060 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:17.447061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.447661 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.447662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:17.448223 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:17.448225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.448552 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.448555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:17.448618 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.448639 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:52:17.461030 (+ 12391us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 14:52:17.461161 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.27.62:39065: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.27.62:39065: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 14:52:17.462146 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13394
2026-08-12T14:52:17Z chronyd exiting
I20260812 14:52:17.514945 4204 test_util.cc:175] -----------------------------------------------
I20260812 14:52:17.515012 4204 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786546313808590-4204-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 14:52:17.515765 4204 mini_oidc.cc:152] Starting JWKS server
I20260812 14:52:17.516366 4204 webserver.cc:460] Webserver started at https://127.0.0.1:44215/ using document root <none> and password file <none>
I20260812 14:52:17.516403 4204 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:52:17.516507 4204 webserver.cc:460] Webserver started at http://127.0.0.1:38979/ using document root <none> and password file <none>
2026-08-12T14:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:17Z Disabled control of system clock
I20260812 14:52:17.524144 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39209
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39209
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/testchainca.cert with env {}
W20260812 14:52:17.602628 13580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:17.602805 13580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:17.602826 13580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:17.603977 13580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:17.604017 13580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:17.604028 13580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:17.604039 13580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:17.604053 13580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:52:17.604068 13580 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:52:17.604079 13580 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44215/jwks.json
I20260812 14:52:17.605530 13580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:17.605705 13580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:17.605851 13580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:17.610294 13599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:17.610414 13594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.610433 13580 server_base.cc:1034] running on GCE node
W20260812 14:52:17.610329 13596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.610740 13580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:17.610991 13580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:17.612118 13580 hybrid_clock.cc:648] HybridClock initialized: now 1786546337612105 us; error 31 us; skew 500 ppm
I20260812 14:52:17.613193 13580 webserver.cc:460] Webserver started at http://127.4.27.62:42037/ using document root <none> and password file <none>
I20260812 14:52:17.613401 13580 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:17.613442 13580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:17.613556 13580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:17.614542 13580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "13975daf93db4eb28dd3aebb77ca755b"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "bd8adad23fb83be5ceb2925edb3ebcc8"
server_key_iv: "d978ae2a90b5b92bb933b1b853f1b776"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.614858 13580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "13975daf93db4eb28dd3aebb77ca755b"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "bd8adad23fb83be5ceb2925edb3ebcc8"
server_key_iv: "d978ae2a90b5b92bb933b1b853f1b776"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.615972 13580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:17.616587 13609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:17.616727 13580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:17.616792 13580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "13975daf93db4eb28dd3aebb77ca755b"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "bd8adad23fb83be5ceb2925edb3ebcc8"
server_key_iv: "d978ae2a90b5b92bb933b1b853f1b776"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.616855 13580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:17.629480 13580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:17.629712 13580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:17.629822 13580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:17.633347 13580 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:39209
I20260812 14:52:17.633385 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:39209 every 8 connection(s)
I20260812 14:52:17.633662 13580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:17.634519 13724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:17.636665 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b: Bootstrap starting.
I20260812 14:52:17.637198 13724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:17.637461 13724 log.cc:824] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:17.637789 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13580
I20260812 14:52:17.637897 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:17.638075 4204 external_mini_cluster.cc:1428] Setting key 97a0f0f8159211cfe498b874f11496e2
I20260812 14:52:17.638123 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b: No bootstrap required, opened a new log
I20260812 14:52:17.640081 13724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13975daf93db4eb28dd3aebb77ca755b" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39209 } }
I20260812 14:52:17.640234 13724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:17.640269 13724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13975daf93db4eb28dd3aebb77ca755b, State: Initialized, Role: FOLLOWER
I20260812 14:52:17.640424 13724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13975daf93db4eb28dd3aebb77ca755b" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39209 } }
I20260812 14:52:17.640515 13724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:17.640547 13724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:17.640709 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:17.641341 13724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13975daf93db4eb28dd3aebb77ca755b" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39209 } }
I20260812 14:52:17.641431 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13975daf93db4eb28dd3aebb77ca755b; no voters:
I20260812 14:52:17.641561 13724 leader_election.cc:288] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:17.641597 13731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:17.641706 13731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [term 1 LEADER]: Becoming Leader. State: Replica: 13975daf93db4eb28dd3aebb77ca755b, State: Running, Role: LEADER
I20260812 14:52:17.641804 13724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:17.641842 13731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13975daf93db4eb28dd3aebb77ca755b" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39209 } }
I20260812 14:52:17.642206 13731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13975daf93db4eb28dd3aebb77ca755b. Latest consensus state: current_term: 1 leader_uuid: "13975daf93db4eb28dd3aebb77ca755b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13975daf93db4eb28dd3aebb77ca755b" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39209 } } }
I20260812 14:52:17.642220 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13975daf93db4eb28dd3aebb77ca755b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13975daf93db4eb28dd3aebb77ca755b" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39209 } } }
I20260812 14:52:17.642373 13731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:17.642383 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:17.642762 13744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:17.643219 13744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:17.645364 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.638539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.4.27.62:39209)
0812 14:52:17.638733 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.638740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.638763 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:52:17.640064 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.640072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.640150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.640273 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.640566 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.642228 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.643433 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.643986 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.644100 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.644720 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.644723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.644724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.644872 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.644906 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.644915 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.644971 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.645095 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.645155 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260812 14:52:17.646271 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:52:17.647089 13744 catalog_manager.cc:1347] Generated new cluster ID: 66e2980459824e5a92fcdc2b48d0a47f
I20260812 14:52:17.647145 13744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:17.665421 13744 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:17.665839 13744 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:17.674871 13744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13975daf93db4eb28dd3aebb77ca755b: Generated new TSK 0
I20260812 14:52:17.675004 13744 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 14:52:17.700309 13772 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 14:52:17.700372 13772 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 14:52:17.700608 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.679395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.4.27.62:39209)
0812 14:52:17.679431 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.679432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.679536 (+ 104us) server_negotiation.cc:407] Connection header received
0812 14:52:17.679601 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.679602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.679613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.679670 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:52:17.679781 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.680239 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.680951 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.681150 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.681203 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.681350 (+ 147us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:52:17.700415 (+ 19065us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 14:52:17.700540 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:52:17.700663 13730 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 14:52:17.700687 13771 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:17.679325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39209 (local address 127.0.0.1:52560)
0812 14:52:17.679476 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.679484 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.679538 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.679667 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.679668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.679670 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:52:17.679725 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:17.679726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.680307 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.680307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:17.680880 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:17.680882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.681209 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.681212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:17.681267 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.681272 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:52:17.700517 (+ 19245us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 14:52:17.700599 (+ 82us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.27.62:39209: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260812 14:52:17.700915 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13580
2026-08-12T14:52:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 14:52:17.749817 4204 mini_oidc.cc:152] Starting JWKS server
I20260812 14:52:17.750478 4204 webserver.cc:460] Webserver started at https://127.0.0.1:33635/ using document root <none> and password file <none>
I20260812 14:52:17.750521 4204 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 14:52:17.750659 4204 webserver.cc:460] Webserver started at http://127.0.0.1:34547/ using document root <none> and password file <none>
2026-08-12T14:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:17Z Disabled control of system clock
I20260812 14:52:17.758129 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:43881
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:33187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:43881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33635/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 14:52:17.835369 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:17.835551 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:17.835580 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:17.836699 13784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:17.836736 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:17.836753 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:17.836769 13784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:17.836786 13784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:52:17.836810 13784 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:52:17.836826 13784 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33635/jwks.json
I20260812 14:52:17.838178 13784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:43881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33635/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:17.838434 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:17.838687 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:17.843108 13794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:17.843115 13792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.843219 13784 server_base.cc:1034] running on GCE node
W20260812 14:52:17.843393 13797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:17.843626 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:17.843906 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:17.845042 13784 hybrid_clock.cc:648] HybridClock initialized: now 1786546337845029 us; error 30 us; skew 500 ppm
I20260812 14:52:17.846140 13784 webserver.cc:460] Webserver started at http://127.4.27.62:41269/ using document root <none> and password file <none>
I20260812 14:52:17.846371 13784 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:17.846427 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:17.846550 13784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:17.847559 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "2b491412b91a41de9589ef027840f904"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "e51bab84c0ca86799c9f4568a9627535"
server_key_iv: "b54a976aabfbb86981b76942d777c0d3"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.847878 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "2b491412b91a41de9589ef027840f904"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "e51bab84c0ca86799c9f4568a9627535"
server_key_iv: "b54a976aabfbb86981b76942d777c0d3"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.849040 13784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:17.849676 13807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:17.849840 13784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:17.849923 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "2b491412b91a41de9589ef027840f904"
format_stamp: "Formatted at 2026-08-12 14:52:17 on dist-test-slave-6zbq"
server_key: "e51bab84c0ca86799c9f4568a9627535"
server_key_iv: "b54a976aabfbb86981b76942d777c0d3"
server_key_version: "encryptionkey@0"
I20260812 14:52:17.849985 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:17.858513 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:17.858750 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:17.858867 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:17.862490 13784 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:43881
I20260812 14:52:17.862519 13903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:43881 every 8 connection(s)
I20260812 14:52:17.862799 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:17.863371 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:17.865752 13904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904: Bootstrap starting.
I20260812 14:52:17.866281 13904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:17.866570 13904 log.cc:824] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:17.867221 13904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904: No bootstrap required, opened a new log
I20260812 14:52:17.868674 13904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b491412b91a41de9589ef027840f904" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43881 } }
I20260812 14:52:17.868799 13904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:17.868834 13904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b491412b91a41de9589ef027840f904, State: Initialized, Role: FOLLOWER
I20260812 14:52:17.868961 13904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b491412b91a41de9589ef027840f904" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43881 } }
I20260812 14:52:17.869019 13904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:17.869040 13904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:17.869083 13904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:17.869554 13904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b491412b91a41de9589ef027840f904" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43881 } }
I20260812 14:52:17.869642 13904 leader_election.cc:302] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b491412b91a41de9589ef027840f904; no voters:
I20260812 14:52:17.869791 13904 leader_election.cc:288] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:17.869838 13909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:17.869987 13909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [term 1 LEADER]: Becoming Leader. State: Replica: 2b491412b91a41de9589ef027840f904, State: Running, Role: LEADER
I20260812 14:52:17.870016 13904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:17.870085 13909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b491412b91a41de9589ef027840f904" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43881 } }
I20260812 14:52:17.870496 13915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b491412b91a41de9589ef027840f904. Latest consensus state: current_term: 1 leader_uuid: "2b491412b91a41de9589ef027840f904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b491412b91a41de9589ef027840f904" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43881 } } }
I20260812 14:52:17.870580 13915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:17.870694 13913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b491412b91a41de9589ef027840f904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b491412b91a41de9589ef027840f904" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43881 } } }
I20260812 14:52:17.870820 13913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:17.871034 13930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:17.871505 13930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:17.871737 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13784
I20260812 14:52:17.871831 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:17.872007 4204 external_mini_cluster.cc:1428] Setting key cf3181aeeae0ac53b6b56f4283485f1f
I20260812 14:52:17.873008 13930 catalog_manager.cc:1347] Generated new cluster ID: 6af189cc2a8a41658dc19187a87a0356
I20260812 14:52:17.873062 13930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:17.877848 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.872338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.4.27.62:43881)
0812 14:52:17.872624 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.872630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.872656 (+ 26us) server_negotiation.cc:407] Connection header received
0812 14:52:17.872772 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.872779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.872845 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.873213 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:17.873295 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.874607 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.875659 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.876229 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.876960 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.877100 (+ 140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:17.877102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:17.877103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:17.877270 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:17.877301 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:17.877313 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:17.877558 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:17.877585 (+ 27us) server_negotiation.cc:299] Negotiation successful
0812 14:52:17.877653 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260812 14:52:17.878732 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:52:17.907033 13930 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:17.907706 13930 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:17.913254 13930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b491412b91a41de9589ef027840f904: Generated new TSK 0
I20260812 14:52:17.913416 13930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:17.926537 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:17.923296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.4.27.62:43881)
0812 14:52:17.923337 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:17.923339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:17.923429 (+ 90us) server_negotiation.cc:407] Connection header received
0812 14:52:17.923494 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:17.923495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:17.923504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:17.923558 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:52:17.923664 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.924101 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.925207 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.925384 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.925426 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.925591 (+ 165us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:52:17.926319 (+ 728us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33635/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 14:52:17.926470 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33630: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33635/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:52:17.926591 13941 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33630: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33635/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 14:52:17.926625 13951 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:17.923219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:43881 (local address 127.0.0.1:33630)
0812 14:52:17.923360 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:17.923370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:17.923433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:17.923558 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:17.923559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:17.923560 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:52:17.923615 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:17.923617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.924207 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.924208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:17.925130 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:17.925134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:17.925445 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:17.925449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:17.925510 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:17.925516 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:52:17.926426 (+ 910us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33635/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 14:52:17.926514 (+ 88us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.27.62:43881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33635/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260812 14:52:17.926810 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13784
E20260812 14:52:17.944633 13953 webserver.cc:533] Webserver: sslize failed:
E20260812 14:52:17.944694 13953 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T14:52:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T14:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:17Z Disabled control of system clock
I20260812 14:52:17.980391 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:43747
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:43747
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/testchainca.cert with env {}
W20260812 14:52:18.058171 13958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:18.058377 13958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:18.058400 13958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:18.059547 13958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:18.059589 13958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:18.059600 13958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:18.059612 13958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:18.059623 13958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 14:52:18.059634 13958 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 14:52:18.059646 13958 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 14:52:18.061136 13958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:43747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:43747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:18.061358 13958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:18.061539 13958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:18.066032 13967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:18.066112 13958 server_base.cc:1034] running on GCE node
W20260812 14:52:18.066042 13970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:18.066042 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:18.066489 13958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:18.066759 13958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:18.067893 13958 hybrid_clock.cc:648] HybridClock initialized: now 1786546338067880 us; error 29 us; skew 500 ppm
I20260812 14:52:18.069003 13958 webserver.cc:460] Webserver started at http://127.4.27.62:38273/ using document root <none> and password file <none>
I20260812 14:52:18.069207 13958 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:18.069255 13958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:18.069377 13958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:18.070333 13958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb"
format_stamp: "Formatted at 2026-08-12 14:52:18 on dist-test-slave-6zbq"
server_key: "c903930688655c596cc6bc7647087395"
server_key_iv: "176c4a540233aa1df8a429eee69bfc4d"
server_key_version: "encryptionkey@0"
I20260812 14:52:18.070689 13958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb"
format_stamp: "Formatted at 2026-08-12 14:52:18 on dist-test-slave-6zbq"
server_key: "c903930688655c596cc6bc7647087395"
server_key_iv: "176c4a540233aa1df8a429eee69bfc4d"
server_key_version: "encryptionkey@0"
I20260812 14:52:18.071846 13958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:18.072495 13982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:18.072659 13958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:18.072742 13958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb"
format_stamp: "Formatted at 2026-08-12 14:52:18 on dist-test-slave-6zbq"
server_key: "c903930688655c596cc6bc7647087395"
server_key_iv: "176c4a540233aa1df8a429eee69bfc4d"
server_key_version: "encryptionkey@0"
I20260812 14:52:18.072806 13958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:18.081372 13958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:18.081598 13958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:18.081710 13958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:18.085228 13958 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:43747
I20260812 14:52:18.085268 14088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:43747 every 8 connection(s)
I20260812 14:52:18.085528 13958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:18.086391 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:18.088429 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb: Bootstrap starting.
I20260812 14:52:18.088917 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:18.089138 14095 log.cc:824] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:18.089697 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb: No bootstrap required, opened a new log
I20260812 14:52:18.091239 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43747 } }
I20260812 14:52:18.091363 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:18.091391 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ae3e3b78ef4c30b8ea21e2489b8ddb, State: Initialized, Role: FOLLOWER
I20260812 14:52:18.091480 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43747 } }
I20260812 14:52:18.091528 14095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:18.091559 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:18.091588 14095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:18.092036 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43747 } }
I20260812 14:52:18.092118 14095 leader_election.cc:302] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46ae3e3b78ef4c30b8ea21e2489b8ddb; no voters:
I20260812 14:52:18.092233 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:18.092278 14098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:18.092415 14098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [term 1 LEADER]: Becoming Leader. State: Replica: 46ae3e3b78ef4c30b8ea21e2489b8ddb, State: Running, Role: LEADER
I20260812 14:52:18.092449 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:18.092515 14098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43747 } }
I20260812 14:52:18.092845 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43747 } } }
I20260812 14:52:18.092924 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:18.092953 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46ae3e3b78ef4c30b8ea21e2489b8ddb. Latest consensus state: current_term: 1 leader_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ae3e3b78ef4c30b8ea21e2489b8ddb" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43747 } } }
I20260812 14:52:18.093035 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:18.093719 14126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:18.094246 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13958
I20260812 14:52:18.094324 14126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:18.094368 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:18.094566 4204 external_mini_cluster.cc:1428] Setting key e329b92ca24f767346ec965c6d2259bf
I20260812 14:52:18.095901 14126 catalog_manager.cc:1347] Generated new cluster ID: 9665b55af25b49a890b185fbc11471b0
I20260812 14:52:18.095949 14126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:18.100050 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:18.094983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.4.27.62:43747)
0812 14:52:18.095159 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:18.095166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:18.095193 (+ 27us) server_negotiation.cc:407] Connection header received
0812 14:52:18.095313 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:18.095319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:18.095368 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:18.095483 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:18.095714 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:18.097015 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:18.098058 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:18.098705 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:18.098821 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:18.099404 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:18.099408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:18.099408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:18.099608 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:18.099647 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:18.099660 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:18.099726 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:18.099781 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 14:52:18.099857 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260812 14:52:18.101058 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:52:18.105407 14126 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:18.105852 14126 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:18.115772 14126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46ae3e3b78ef4c30b8ea21e2489b8ddb: Generated new TSK 0
I20260812 14:52:18.115908 14126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:18.144323 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:18.124441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.4.27.62:43747)
0812 14:52:18.124488 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:18.124490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:18.124582 (+ 92us) server_negotiation.cc:407] Connection header received
0812 14:52:18.124650 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:18.124650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:18.124660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:18.124711 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0812 14:52:18.124826 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:18.125304 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:18.126420 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:18.126650 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:18.126712 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:18.126907 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 14:52:18.144153 (+ 17246us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 14:52:18.144275 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53518: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260812 14:52:18.144374 14142 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53518: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 14:52:18.144539 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:18.124376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:43747 (local address 127.0.0.1:53518)
0812 14:52:18.124520 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:18.124530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:18.124590 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:18.124721 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:18.124722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:18.124723 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0812 14:52:18.124771 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:18.124773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:18.125363 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:18.125363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:18.126301 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:18.126304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:18.126723 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:18.126727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:18.126788 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:18.126793 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 14:52:18.144259 (+ 17466us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 14:52:18.144366 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.27.62:43747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260812 14:52:18.144999 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13958
2026-08-12T14:52:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (199 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:18 dist-test-slave-6zbq krb5kdc[14174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:18 dist-test-slave-6zbq krb5kdc[14174](info): set up 2 sockets
Aug 12 14:52:18 dist-test-slave-6zbq krb5kdc[14174](info): commencing operation
krb5kdc: starting...
W20260812 14:52:19.493010 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:19 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:19.598335 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:38647
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:19.675209 14193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:19.675387 14193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:19.675415 14193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:19.676510 14193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:19.676548 14193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:19.676566 14193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:19.676582 14193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:19.676599 14193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:19.677996 14193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:19.678215 14193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:19.678431 14193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:19.682843 14207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:19.682879 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:19.683022 14193 server_base.cc:1034] running on GCE node
W20260812 14:52:19.683120 14205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:19.683346 14193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:19.683591 14193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:19.684736 14193 hybrid_clock.cc:648] HybridClock initialized: now 1786546339684717 us; error 36 us; skew 500 ppm
Aug 12 14:52:19 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:19.687059 14193 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:19.687467 14193 webserver.cc:460] Webserver started at http://127.4.27.62:37623/ using document root <none> and password file <none>
I20260812 14:52:19.687688 14193 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:19.687737 14193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:19.687857 14193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:19.695194 14193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "b702537185e149de8a8b461c67d93f71"
format_stamp: "Formatted at 2026-08-12 14:52:19 on dist-test-slave-6zbq"
server_key: "57b31ea63bd913809d512899487e3cc5"
server_key_iv: "98ae8ae72385fa2ce9283cf0c551f719"
server_key_version: "encryptionkey@0"
I20260812 14:52:19.695492 14193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "b702537185e149de8a8b461c67d93f71"
format_stamp: "Formatted at 2026-08-12 14:52:19 on dist-test-slave-6zbq"
server_key: "57b31ea63bd913809d512899487e3cc5"
server_key_iv: "98ae8ae72385fa2ce9283cf0c551f719"
server_key_version: "encryptionkey@0"
I20260812 14:52:19.696647 14193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:19.697263 14220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:19.697408 14193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:19.697489 14193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "b702537185e149de8a8b461c67d93f71"
format_stamp: "Formatted at 2026-08-12 14:52:19 on dist-test-slave-6zbq"
server_key: "57b31ea63bd913809d512899487e3cc5"
server_key_iv: "98ae8ae72385fa2ce9283cf0c551f719"
server_key_version: "encryptionkey@0"
I20260812 14:52:19.697547 14193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:19.721333 14193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:19.723239 14193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:19.723364 14193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:19.727011 14193 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:38647
I20260812 14:52:19.727051 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:38647 every 8 connection(s)
I20260812 14:52:19.727350 14193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:19.728088 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:19.730129 14316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71: Bootstrap starting.
I20260812 14:52:19.730697 14316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:19.730954 14316 log.cc:824] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:19.731539 14316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71: No bootstrap required, opened a new log
I20260812 14:52:19.732259 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 14193
I20260812 14:52:19.732349 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:19.732506 4204 external_mini_cluster.cc:1428] Setting key 7d99348c11f339aab77b02b3625416ef
I20260812 14:52:19.733193 14316 raft_consensus.cc:348] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702537185e149de8a8b461c67d93f71" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38647 } }
I20260812 14:52:19.733330 14316 raft_consensus.cc:374] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:19.733363 14316 raft_consensus.cc:729] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b702537185e149de8a8b461c67d93f71, State: Initialized, Role: FOLLOWER
I20260812 14:52:19.733481 14316 consensus_queue.cc:260] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702537185e149de8a8b461c67d93f71" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38647 } }
I20260812 14:52:19.733556 14316 raft_consensus.cc:388] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:19.733587 14316 raft_consensus.cc:482] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:19.733629 14316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:19.734103 14316 raft_consensus.cc:504] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702537185e149de8a8b461c67d93f71" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38647 } }
I20260812 14:52:19.734196 14316 leader_election.cc:302] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b702537185e149de8a8b461c67d93f71; no voters:
I20260812 14:52:19.734383 14316 leader_election.cc:288] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:19.734414 14323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:19.734545 14323 raft_consensus.cc:686] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [term 1 LEADER]: Becoming Leader. State: Replica: b702537185e149de8a8b461c67d93f71, State: Running, Role: LEADER
I20260812 14:52:19.734632 14316 sys_catalog.cc:564] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:19.734633 14323 consensus_queue.cc:237] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702537185e149de8a8b461c67d93f71" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38647 } }
I20260812 14:52:19.735100 14329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b702537185e149de8a8b461c67d93f71. Latest consensus state: current_term: 1 leader_uuid: "b702537185e149de8a8b461c67d93f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702537185e149de8a8b461c67d93f71" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38647 } } }
I20260812 14:52:19.735117 14328 sys_catalog.cc:455] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b702537185e149de8a8b461c67d93f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b702537185e149de8a8b461c67d93f71" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38647 } } }
I20260812 14:52:19.735270 14329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:19.735282 14328 sys_catalog.cc:458] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:19.735882 14341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:19.736308 14341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:19.738040 14341 catalog_manager.cc:1347] Generated new cluster ID: 627a09d826af4375bcc33f749f8afa27
I20260812 14:52:19.738093 14341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:52:19 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:19.742601 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:19.732962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.4.27.62:38647)
0812 14:52:19.733120 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:19.733129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:19.733198 (+ 69us) server_negotiation.cc:407] Connection header received
0812 14:52:19.733335 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:19.733341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:19.733384 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:19.733511 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:19.734177 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:19.735457 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:19.736385 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:19.736828 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:19.737206 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:19.739319 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:19.739323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:19.739324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:19.739335 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:19.740747 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:19.741060 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:19.741063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:19.741063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:19.741180 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:19.741360 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:19.741361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:19.741361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:19.741584 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:19.742254 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:19.742324 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:52:19.742461 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260812 14:52:19.750880 14341 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:19.751354 14341 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:19.753896 14341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b702537185e149de8a8b461c67d93f71: Generated new TSK 0
I20260812 14:52:19.754046 14341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:19.778558 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38647
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:19.857201 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:19.857376 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:19.857398 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:19.858505 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:19.858542 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:19.858567 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:19.859995 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:19.860170 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:19.860332 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:19.864809 14372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:19.864926 14363 server_base.cc:1034] running on GCE node
W20260812 14:52:19.864802 14377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:19.864802 14374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:19.865275 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:19.865492 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:19.866618 14363 hybrid_clock.cc:648] HybridClock initialized: now 1786546339866611 us; error 35 us; skew 500 ppm
Aug 12 14:52:19 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:19.868873 14363 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:19.869266 14363 webserver.cc:460] Webserver started at http://127.4.27.1:35769/ using document root <none> and password file <none>
I20260812 14:52:19.869480 14363 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:19.869527 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:19.869644 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:19.870658 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "fe84d17040af40fd94f65c771f135b87"
format_stamp: "Formatted at 2026-08-12 14:52:19 on dist-test-slave-6zbq"
server_key: "af1c3b33749789d23b0a4a6dcd62a4a2"
server_key_iv: "065549cd0fc5f3d1b757afcc60573d10"
server_key_version: "encryptionkey@0"
I20260812 14:52:19.870959 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "fe84d17040af40fd94f65c771f135b87"
format_stamp: "Formatted at 2026-08-12 14:52:19 on dist-test-slave-6zbq"
server_key: "af1c3b33749789d23b0a4a6dcd62a4a2"
server_key_iv: "065549cd0fc5f3d1b757afcc60573d10"
server_key_version: "encryptionkey@0"
I20260812 14:52:19.872092 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:19.872727 14399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:19.872866 14363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:19.872946 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "fe84d17040af40fd94f65c771f135b87"
format_stamp: "Formatted at 2026-08-12 14:52:19 on dist-test-slave-6zbq"
server_key: "af1c3b33749789d23b0a4a6dcd62a4a2"
server_key_iv: "065549cd0fc5f3d1b757afcc60573d10"
server_key_version: "encryptionkey@0"
I20260812 14:52:19.873003 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:19.880045 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:19.881606 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:19.881724 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:19.881908 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:19.882171 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:19.882205 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:19.882230 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:19.882254 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:19.887586 14363 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:40913
I20260812 14:52:19.887619 14620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:40913 every 8 connection(s)
I20260812 14:52:19.887881 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:19.892448 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 14363
I20260812 14:52:19.892531 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:19.892690 4204 external_mini_cluster.cc:1428] Setting key 853611195ebda3f811206047e7488e88
Aug 12 14:52:19 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546339, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:19.896402 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:19.888507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:56811 (local address 127.4.27.62:38647)
0812 14:52:19.888563 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:19.888565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:19.888738 (+ 173us) server_negotiation.cc:407] Connection header received
0812 14:52:19.889556 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:19.889557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:19.889565 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:19.889613 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:19.890122 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:19.890656 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:19.891550 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:19.891733 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:19.891772 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:19.893781 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:19.893782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:19.893782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:19.893789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:19.895210 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:19.895563 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:19.895564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:19.895564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:19.895625 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:19.895823 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:19.895824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:19.895825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:19.895959 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:19.896028 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:19.896229 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 14:52:19.896321 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:52:19.896443 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:19.888476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38647 (local address 127.4.27.1:56811)
0812 14:52:19.888626 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:19.888664 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:19.889480 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:19.889663 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:19.889666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:19.889890 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:19.890057 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:19.890061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:19.890719 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:19.890719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:19.891486 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:19.891488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:19.891781 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:19.891782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:19.891875 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:19.892225 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:19.892232 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:19.893667 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:19.895292 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:19.895298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:19.895300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:19.895499 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:19.895687 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:19.895687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:19.895688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:19.895772 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:19.896016 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:19.896019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:19.896117 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 14:52:19.896193 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:52:19.896301 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260812 14:52:19.896862 14622 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38647
I20260812 14:52:19.896958 14622 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:19.897128 14622 heartbeater.cc:511] Master 127.4.27.62:38647 requested a full tablet report, sending...
I20260812 14:52:19.897658 14248 ts_manager.cc:194] Registered new tserver with Master: fe84d17040af40fd94f65c771f135b87 (127.4.27.1:40913)
I20260812 14:52:19.898559 14248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:56811
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:19.915292 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38647
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:19.992578 14632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:19.992765 14632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:19.992787 14632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:19.993857 14632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:19.993888 14632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:19.993913 14632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:19.995342 14632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:19.995510 14632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:19.995673 14632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:52:20.000264 14632 server_base.cc:1034] running on GCE node
W20260812 14:52:20.000205 14644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:20.000203 14641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:20.000200 14640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:20.000607 14632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:20.000834 14632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:20.001959 14632 hybrid_clock.cc:648] HybridClock initialized: now 1786546340001952 us; error 26 us; skew 500 ppm
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.004199 14632 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:20.004558 14632 webserver.cc:460] Webserver started at http://127.4.27.2:44519/ using document root <none> and password file <none>
I20260812 14:52:20.004715 14632 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:20.004755 14632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:20.004841 14632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:20.005734 14632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "502fd201a89e4c6db8b9de00c6e31c82"
format_stamp: "Formatted at 2026-08-12 14:52:20 on dist-test-slave-6zbq"
server_key: "77afae58be38c063522203188dc16e01"
server_key_iv: "9cbf77dd88ce04cb5eea1bd002c39400"
server_key_version: "encryptionkey@0"
I20260812 14:52:20.006011 14632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "502fd201a89e4c6db8b9de00c6e31c82"
format_stamp: "Formatted at 2026-08-12 14:52:20 on dist-test-slave-6zbq"
server_key: "77afae58be38c063522203188dc16e01"
server_key_iv: "9cbf77dd88ce04cb5eea1bd002c39400"
server_key_version: "encryptionkey@0"
I20260812 14:52:20.007180 14632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:52:20.007771 14658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:20.007915 14632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:20.007992 14632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "502fd201a89e4c6db8b9de00c6e31c82"
format_stamp: "Formatted at 2026-08-12 14:52:20 on dist-test-slave-6zbq"
server_key: "77afae58be38c063522203188dc16e01"
server_key_iv: "9cbf77dd88ce04cb5eea1bd002c39400"
server_key_version: "encryptionkey@0"
I20260812 14:52:20.008056 14632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:20.031551 14632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:20.033428 14632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:20.033550 14632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:20.033740 14632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:20.034001 14632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:20.034032 14632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:20.034051 14632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:20.034073 14632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:20.039351 14632 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:35337
I20260812 14:52:20.039392 14873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:35337 every 8 connection(s)
I20260812 14:52:20.039649 14632 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:20.047426 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.040260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:32901 (local address 127.4.27.62:38647)
0812 14:52:20.040328 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.040331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.040475 (+ 144us) server_negotiation.cc:407] Connection header received
0812 14:52:20.041309 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.041310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.041321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.041355 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.041866 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.042389 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.043265 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.043442 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.043482 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.045224 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:20.045226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:20.045226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:20.045232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:20.046481 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.046785 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.046786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.046786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.046830 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.046992 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.046993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.046993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.047082 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:20.047124 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:20.047301 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:52:20.047363 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:52:20.047565 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.040227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38647 (local address 127.4.27.2:32901)
0812 14:52:20.040369 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:20.040400 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:20.041222 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:20.041410 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:20.041413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:20.041626 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:20.041808 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:20.041812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.042452 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.042453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:20.043202 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:20.043204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.043490 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.043491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:20.043576 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.043926 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:20.043933 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:20.045149 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:20.046538 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:20.046541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:20.046542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:20.046736 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:20.046871 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:20.046872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:20.046872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:20.046953 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:20.047121 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:20.047124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:20.047220 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 14:52:20.047289 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:52:20.047391 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260812 14:52:20.047996 14874 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38647
I20260812 14:52:20.048076 14874 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:20.048235 14874 heartbeater.cc:511] Master 127.4.27.62:38647 requested a full tablet report, sending...
I20260812 14:52:20.048589 14248 ts_manager.cc:194] Registered new tserver with Master: 502fd201a89e4c6db8b9de00c6e31c82 (127.4.27.2:35337)
I20260812 14:52:20.048931 14248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:32901
I20260812 14:52:20.049466 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 14632
I20260812 14:52:20.049564 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:20.049734 4204 external_mini_cluster.cc:1428] Setting key 5d8584729412ea4978082932a7eb442b
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:20.070993 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38647
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:20.148860 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:20.149036 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:20.149057 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:20.150197 14882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:20.150244 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:20.150271 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:20.151705 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:20.151909 14882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:20.152102 14882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:52:20.156682 14882 server_base.cc:1034] running on GCE node
W20260812 14:52:20.156656 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:20.156654 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:20.156713 14891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:20.157034 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:20.157263 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:20.158386 14882 hybrid_clock.cc:648] HybridClock initialized: now 1786546340158372 us; error 27 us; skew 500 ppm
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.160678 14882 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:20.161069 14882 webserver.cc:460] Webserver started at http://127.4.27.3:41557/ using document root <none> and password file <none>
I20260812 14:52:20.161249 14882 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:20.161286 14882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:20.161371 14882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:20.162288 14882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "44c6c558d597452286ea09f0e219f3d2"
format_stamp: "Formatted at 2026-08-12 14:52:20 on dist-test-slave-6zbq"
server_key: "89a96b8b8fdb9d312c22c4b3cd5871a0"
server_key_iv: "70c9259d0b98a8b9962ce5a3554894f7"
server_key_version: "encryptionkey@0"
I20260812 14:52:20.162617 14882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "44c6c558d597452286ea09f0e219f3d2"
format_stamp: "Formatted at 2026-08-12 14:52:20 on dist-test-slave-6zbq"
server_key: "89a96b8b8fdb9d312c22c4b3cd5871a0"
server_key_iv: "70c9259d0b98a8b9962ce5a3554894f7"
server_key_version: "encryptionkey@0"
I20260812 14:52:20.163726 14882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:20.164311 14908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:20.164448 14882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:20.164534 14882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "44c6c558d597452286ea09f0e219f3d2"
format_stamp: "Formatted at 2026-08-12 14:52:20 on dist-test-slave-6zbq"
server_key: "89a96b8b8fdb9d312c22c4b3cd5871a0"
server_key_iv: "70c9259d0b98a8b9962ce5a3554894f7"
server_key_version: "encryptionkey@0"
I20260812 14:52:20.164595 14882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:20.176067 14882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:20.177726 14882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:20.177820 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:20.177978 14882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:20.178197 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:20.178226 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:20.178246 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:20.178261 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:20.183634 14882 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:39109
I20260812 14:52:20.183697 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:39109 every 8 connection(s)
I20260812 14:52:20.183945 14882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:20.184840 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 14882
I20260812 14:52:20.184939 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:20.185130 4204 external_mini_cluster.cc:1428] Setting key a38341a1a5f1b71b0608ee99e7725b8a
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:20.191828 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.184546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:44999 (local address 127.4.27.62:38647)
0812 14:52:20.184590 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.184592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.184760 (+ 168us) server_negotiation.cc:407] Connection header received
0812 14:52:20.185764 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.185765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.185776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.185828 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.186410 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.186897 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.187780 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.187953 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.187996 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.189666 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:20.189667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:20.189668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:20.189675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:20.190884 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.191179 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.191179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.191180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.191232 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.191395 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.191395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.191396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.191483 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:20.191525 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:20.191715 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 14:52:20.191768 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:20.191968 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.184507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38647 (local address 127.4.27.3:44999)
0812 14:52:20.184640 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:20.184684 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:20.185689 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:20.185875 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:20.185877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:20.186127 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:20.186297 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:20.186301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.186961 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.186962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:20.187723 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:20.187725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.188001 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.188002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:20.188086 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.188414 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:20.188422 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:20.189590 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:20.190940 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:20.190943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:20.190945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:20.191127 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:20.191273 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:20.191273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:20.191273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:20.191355 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:20.191523 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:20.191525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:20.191621 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 14:52:20.191693 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:52:20.191797 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260812 14:52:20.192384 15125 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38647
I20260812 14:52:20.192464 15125 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:20.192589 15125 heartbeater.cc:511] Master 127.4.27.62:38647 requested a full tablet report, sending...
I20260812 14:52:20.192894 14248 ts_manager.cc:194] Registered new tserver with Master: 44c6c558d597452286ea09f0e219f3d2 (127.4.27.3:39109)
I20260812 14:52:20.193228 14248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:44999
I20260812 14:52:20.196640 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/test-admin.keytab.
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.209826 4204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 14:52:20.211282 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:20.210598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.4.27.1:40913)
0812 14:52:20.210746 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.210750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.210759 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:20.210782 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.210783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.210795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.210855 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.211210 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36840: BlockingRecv error: recv got EOF from 127.0.0.1:36840 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260812 14:52:20.211364 4204 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.212230 4204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.212984 4204 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:52:20.219466 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.213431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.4.27.1:40913)
0812 14:52:20.213478 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.213480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.213488 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:20.213538 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.213539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.213548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.213590 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.213917 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.214638 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.215286 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.215563 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.215612 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.217349 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:20.217356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:20.217356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:20.217361 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:20.218625 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.218897 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.218898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.218899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.218944 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.219060 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.219060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.219060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.219175 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:20.219261 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:20.219319 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 14:52:20.219378 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260812 14:52:20.220796 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:20.220417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.4.27.1:40913)
0812 14:52:20.220468 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.220469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.220476 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:20.220517 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.220518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.220526 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.220565 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.220762 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36856: BlockingRecv error: recv got EOF from 127.0.0.1:36856 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260812 14:52:20.220878 4204 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.221689 4204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.222496 4204 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:52:20.228189 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.222907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.4.27.1:40913)
0812 14:52:20.222954 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.222956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.222987 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:52:20.223031 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.223032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.223039 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.223078 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.223367 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.223802 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.224454 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.224636 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.224673 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.226301 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:20.226302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:20.226302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:20.226307 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:20.227454 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.227703 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.227703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.227704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.227750 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.227863 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.227863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.227863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.227931 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:20.227988 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:20.228068 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 14:52:20.228120 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260812 14:52:20.229569 15141 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:20.229030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.4.27.1:40913)
0812 14:52:20.229065 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.229066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.229106 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:52:20.229218 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.229219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.229228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.229283 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.229532 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36874: BlockingRecv error: recv got EOF from 127.0.0.1:36874 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:52:20.229799 4204 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786546313808590-4204-0/krb5kdc/krb5cc' of type FILE
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.230665 4204 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:20.231402 4204 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[14174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:52:20.237393 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:20.231823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.4.27.1:40913)
0812 14:52:20.231866 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:20.231868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:20.231889 (+ 21us) server_negotiation.cc:407] Connection header received
0812 14:52:20.231959 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:20.231960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:20.231967 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:20.232002 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:20.232338 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.232765 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.233376 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:20.233552 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:20.233609 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:20.235384 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:20.235385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:20.235385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:20.235392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:20.236499 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.236779 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.236780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.236780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.236832 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:20.236986 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:20.236986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:20.236987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:20.237072 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:20.237147 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:20.237247 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 14:52:20.237316 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:52:20.237888 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 14363
I20260812 14:52:20.258479 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 14632
I20260812 14:52:20.280217 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 14882
I20260812 14:52:20.301661 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 14193
2026-08-12T14:52:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1979 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[15164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[15164](info): set up 2 sockets
Aug 12 14:52:20 dist-test-slave-6zbq krb5kdc[15164](info): commencing operation
krb5kdc: starting...
W20260812 14:52:21.479991 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:21 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab.
Entry for principal oryx/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab.
I20260812 14:52:21.588907 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46817
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab
--principal=oryx/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:21.667667 15187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:21.667838 15187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:21.667858 15187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:21.668951 15187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:21.668988 15187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:21.669001 15187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:21.669024 15187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:21.669047 15187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:21.670521 15187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab
--principal=oryx/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:21.670730 15187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:21.670922 15187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:21.675439 15207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:21.675436 15201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:21.675491 15187 server_base.cc:1034] running on GCE node
W20260812 14:52:21.675446 15198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:21.675863 15187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:21.676133 15187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:21.677269 15187 hybrid_clock.cc:648] HybridClock initialized: now 1786546341677256 us; error 30 us; skew 500 ppm
Aug 12 14:52:21 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:21.683583 15187 init.cc:377] Logged in from keytab as oryx/127.4.27.62@KRBTEST.COM (short username oryx)
I20260812 14:52:21.683977 15187 webserver.cc:460] Webserver started at http://127.4.27.62:46423/ using document root <none> and password file <none>
I20260812 14:52:21.684196 15187 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:21.684244 15187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:21.684365 15187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:21.685310 15187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "129addd4a43b4f1b96b7b060cb647c6c"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "54d4d9de02d0345de9fd8154f3ea33f8"
server_key_iv: "57454fa76956d55a16c8e51c3909bf10"
server_key_version: "encryptionkey@0"
I20260812 14:52:21.685614 15187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "129addd4a43b4f1b96b7b060cb647c6c"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "54d4d9de02d0345de9fd8154f3ea33f8"
server_key_iv: "57454fa76956d55a16c8e51c3909bf10"
server_key_version: "encryptionkey@0"
I20260812 14:52:21.686864 15187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:21.687508 15217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:21.687649 15187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:21.687729 15187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "129addd4a43b4f1b96b7b060cb647c6c"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "54d4d9de02d0345de9fd8154f3ea33f8"
server_key_iv: "57454fa76956d55a16c8e51c3909bf10"
server_key_version: "encryptionkey@0"
I20260812 14:52:21.687786 15187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:21.708150 15187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:21.709743 15187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:21.709859 15187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:21.713567 15187 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46817
I20260812 14:52:21.713608 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46817 every 8 connection(s)
I20260812 14:52:21.713883 15187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:21.714790 15313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:21.716832 15313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Bootstrap starting.
I20260812 14:52:21.717371 15313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:21.717617 15313 log.cc:824] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:21.718216 15313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: No bootstrap required, opened a new log
I20260812 14:52:21.719727 15313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } }
I20260812 14:52:21.719853 15313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:21.719885 15313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129addd4a43b4f1b96b7b060cb647c6c, State: Initialized, Role: FOLLOWER
I20260812 14:52:21.720014 15313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } }
I20260812 14:52:21.720072 15313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:21.720103 15313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:21.720147 15313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:21.720866 15313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } }
I20260812 14:52:21.720953 15313 leader_election.cc:302] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129addd4a43b4f1b96b7b060cb647c6c; no voters:
I20260812 14:52:21.721101 15313 leader_election.cc:288] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:21.721134 15319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:21.721282 15319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 LEADER]: Becoming Leader. State: Replica: 129addd4a43b4f1b96b7b060cb647c6c, State: Running, Role: LEADER
I20260812 14:52:21.721336 15313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:21.721369 15319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } }
I20260812 14:52:21.721716 15323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129addd4a43b4f1b96b7b060cb647c6c. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } }
I20260812 14:52:21.721805 15323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:21.721710 15322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } }
I20260812 14:52:21.722018 15322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:21.722240 15342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:21.722725 15342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:21.723274 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 15187
I20260812 14:52:21.723392 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:21.723587 4204 external_mini_cluster.cc:1428] Setting key 7efef3f428fa1e77c3d7ab7ed9c019d2
I20260812 14:52:21.724339 15342 catalog_manager.cc:1347] Generated new cluster ID: 16d04f3c2f684628800b7ec91f2a8a6d
I20260812 14:52:21.724399 15342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:52:21 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:21.734676 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:21.724024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.4.27.62:46817)
0812 14:52:21.724577 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:21.724586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:21.724616 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:52:21.724722 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:21.724729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:21.724779 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:21.724909 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:21.725620 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:21.727009 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:21.727987 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:21.728592 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:21.728706 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:21.731332 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:21.731336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:21.731337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:21.731348 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:21.733166 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:21.733536 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:21.733539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:21.733539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:21.733632 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:21.733817 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:21.733817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:21.733818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:21.734090 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:21.734230 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:21.734316 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 14:52:21.734464 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":67,"threads_started":1}
I20260812 14:52:21.738658 15342 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:21.739112 15342 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:21.754819 15342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Generated new TSK 0
I20260812 14:52:21.755029 15342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab.
Entry for principal oryx/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab.
I20260812 14:52:21.780757 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46817
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:21.858778 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:21.858954 15363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:21.858975 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:21.860067 15363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:21.860105 15363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:21.860133 15363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:21.861524 15363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:21.861737 15363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:21.861931 15363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:21.866447 15383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:21.866474 15389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:21.866477 15363 server_base.cc:1034] running on GCE node
W20260812 14:52:21.866633 15379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:21.866807 15363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:21.867026 15363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:21.868165 15363 hybrid_clock.cc:648] HybridClock initialized: now 1786546341868149 us; error 28 us; skew 500 ppm
Aug 12 14:52:21 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:21.870611 15363 init.cc:377] Logged in from keytab as oryx/127.4.27.1@KRBTEST.COM (short username oryx)
I20260812 14:52:21.871029 15363 webserver.cc:460] Webserver started at http://127.4.27.1:41171/ using document root <none> and password file <none>
I20260812 14:52:21.871222 15363 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:21.871269 15363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:21.871397 15363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:21.872364 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "3db8e50d4dcf4369bc5473ee39163e7a"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "15b236dac4864d0e85f714624545c40e"
server_key_iv: "2b328e7306f1923ac8f66c862fe2caef"
server_key_version: "encryptionkey@0"
I20260812 14:52:21.872685 15363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "3db8e50d4dcf4369bc5473ee39163e7a"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "15b236dac4864d0e85f714624545c40e"
server_key_iv: "2b328e7306f1923ac8f66c862fe2caef"
server_key_version: "encryptionkey@0"
I20260812 14:52:21.873914 15363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:21.874619 15397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:21.874768 15363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:21.874850 15363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "3db8e50d4dcf4369bc5473ee39163e7a"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "15b236dac4864d0e85f714624545c40e"
server_key_iv: "2b328e7306f1923ac8f66c862fe2caef"
server_key_version: "encryptionkey@0"
I20260812 14:52:21.874917 15363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:21.889139 15363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:21.890954 15363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:21.891080 15363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:21.891287 15363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:21.891582 15363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:21.891616 15363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:21.891644 15363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:21.891669 15363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:21.897336 15363 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:33559
I20260812 14:52:21.897400 15583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:33559 every 8 connection(s)
I20260812 14:52:21.897680 15363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:21 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:21.904865 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 15363
I20260812 14:52:21.904968 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:21.906198 4204 external_mini_cluster.cc:1428] Setting key 3f981cf0eeac6724afdd3e486f6fee24
I20260812 14:52:21.907110 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:21.898312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:49319 (local address 127.4.27.62:46817)
0812 14:52:21.898386 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:21.898388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:21.898558 (+ 170us) server_negotiation.cc:407] Connection header received
0812 14:52:21.899066 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:21.899068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:21.899085 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:21.899137 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:21.900048 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:21.900548 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:21.901500 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:21.901709 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:21.901765 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:21.903609 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:21.903611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:21.903611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:21.903619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:21.904657 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:21.905024 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:21.905026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:21.905026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:21.905098 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:21.906224 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:21.906228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:21.906228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:21.906555 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:21.906635 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:21.906920 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 14:52:21.907020 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:52:21.907274 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:21.898271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:49319)
0812 14:52:21.898436 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:21.898479 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:21.898968 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:21.899205 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:21.899209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:21.899768 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:21.899949 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:21.899954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:21.900630 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:21.900631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:21.901413 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:21.901415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:21.901782 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:21.901784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:21.901899 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:21.902285 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:21.902292 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:21.903521 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:21.904731 (+ 1210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:21.904746 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:21.904748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:21.904954 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:21.905386 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:21.905387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:21.905387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:21.905495 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:21.906640 (+ 1145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:21.906648 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:21.906796 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 14:52:21.906884 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:52:21.907037 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260812 14:52:21.907774 15586 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46817
I20260812 14:52:21.907869 15586 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:21.908043 15586 heartbeater.cc:511] Master 127.4.27.62:46817 requested a full tablet report, sending...
I20260812 14:52:21.908514 15247 ts_manager.cc:194] Registered new tserver with Master: 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559)
I20260812 14:52:21.909112 15247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:49319
WARNING: no policy specified for oryx/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab.
Entry for principal oryx/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab.
I20260812 14:52:21.928537 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46817
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:22.007633 15593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:22.007896 15593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:22.007933 15593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:22.009083 15593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:22.009122 15593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:22.009146 15593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:22.010634 15593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:22.010841 15593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:22.011034 15593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.015594 15609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.015595 15604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.015697 15593 server_base.cc:1034] running on GCE node
W20260812 14:52:22.015631 15603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.015961 15593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.016188 15593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.017329 15593 hybrid_clock.cc:648] HybridClock initialized: now 1786546342017316 us; error 31 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.019870 15593 init.cc:377] Logged in from keytab as oryx/127.4.27.2@KRBTEST.COM (short username oryx)
I20260812 14:52:22.020347 15593 webserver.cc:460] Webserver started at http://127.4.27.2:43271/ using document root <none> and password file <none>
I20260812 14:52:22.020541 15593 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.020581 15593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.020684 15593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:22.021745 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "b7a80beb19f348b8a7835bf593fa8c2b"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "bc477a496085e48198428f0b74336916"
server_key_iv: "b6e53ccb169e0c427da7ddc24ebbdd47"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.022079 15593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "b7a80beb19f348b8a7835bf593fa8c2b"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "bc477a496085e48198428f0b74336916"
server_key_iv: "b6e53ccb169e0c427da7ddc24ebbdd47"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.023396 15593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:22.024078 15625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.024259 15593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:22.024353 15593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "b7a80beb19f348b8a7835bf593fa8c2b"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "bc477a496085e48198428f0b74336916"
server_key_iv: "b6e53ccb169e0c427da7ddc24ebbdd47"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.024412 15593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.038547 15593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.040180 15593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:22.040311 15593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.040508 15593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:22.040792 15593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:22.040830 15593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.040859 15593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:22.040877 15593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.046269 15593 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36901
I20260812 14:52:22.046312 15832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36901 every 8 connection(s)
I20260812 14:52:22.046627 15593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:22.053046 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 15593
I20260812 14:52:22.053153 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:22.053357 4204 external_mini_cluster.cc:1428] Setting key 966d50634aafceabb268a5215e19433c
I20260812 14:52:22.055850 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.047409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:38509 (local address 127.4.27.62:46817)
0812 14:52:22.047487 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.047489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.047643 (+ 154us) server_negotiation.cc:407] Connection header received
0812 14:52:22.048229 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.048230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.048244 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.048292 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.049206 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.049695 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.050675 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.050870 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.050926 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.053032 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.053034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.053034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.053042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.054604 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.055046 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.055047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.055047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.055092 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.055304 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.055305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.055305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.055396 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.055439 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.055714 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.055787 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:52:22.056077 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.047357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:38509)
0812 14:52:22.047516 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.047564 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.048061 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.048367 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.048370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.048923 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.049111 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.049116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.049775 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.049776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.050599 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.050602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.050938 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.050939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.051048 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.051474 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.051482 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.052918 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.054676 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.054684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.054686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.054975 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.055132 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.055133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.055133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.055255 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.055446 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.055451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.055598 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.055711 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.055866 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260812 14:52:22.056609 15833 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46817
I20260812 14:52:22.056713 15833 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.056891 15833 heartbeater.cc:511] Master 127.4.27.62:46817 requested a full tablet report, sending...
I20260812 14:52:22.057240 15247 ts_manager.cc:194] Registered new tserver with Master: b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:22.057592 15247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:38509
WARNING: no policy specified for oryx/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab.
Entry for principal oryx/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab.
I20260812 14:52:22.076900 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46817
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:22.154148 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:22.154328 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:22.154374 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:22.155454 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:22.155493 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:22.155526 15841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:22.156912 15841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:22.157130 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:22.157323 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.161799 15853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.161787 15852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.161775 15857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.161845 15841 server_base.cc:1034] running on GCE node
I20260812 14:52:22.162104 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.162299 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.163442 15841 hybrid_clock.cc:648] HybridClock initialized: now 1786546342163433 us; error 26 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.165627 15841 init.cc:377] Logged in from keytab as oryx/127.4.27.3@KRBTEST.COM (short username oryx)
I20260812 14:52:22.166013 15841 webserver.cc:460] Webserver started at http://127.4.27.3:38417/ using document root <none> and password file <none>
I20260812 14:52:22.166200 15841 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.166246 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.166399 15841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:22.167344 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "dfd3cd8bc4464b4088952aef04a0d5df"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "d18492160bd9ae3ea11670d4eceaa602"
server_key_iv: "ef5714f0f26eac4a0a4e3abd0b50d3ff"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.167657 15841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "dfd3cd8bc4464b4088952aef04a0d5df"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "d18492160bd9ae3ea11670d4eceaa602"
server_key_iv: "ef5714f0f26eac4a0a4e3abd0b50d3ff"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.168805 15841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:22.169416 15867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.169559 15841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:22.169638 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "dfd3cd8bc4464b4088952aef04a0d5df"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "d18492160bd9ae3ea11670d4eceaa602"
server_key_iv: "ef5714f0f26eac4a0a4e3abd0b50d3ff"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.169701 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.179013 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.180953 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:22.181080 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.181294 15841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:22.181573 15841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:22.181607 15841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.181632 15841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:22.181650 15841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.187026 15841 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42207
I20260812 14:52:22.187080 16072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42207 every 8 connection(s)
I20260812 14:52:22.187346 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:22.190748 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 15841
I20260812 14:52:22.190847 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:22.191026 4204 external_mini_cluster.cc:1428] Setting key fbaeb83c21f384148b3c5afec6c08c28
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:22.196411 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.187977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:51965 (local address 127.4.27.62:46817)
0812 14:52:22.188037 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.188039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.188226 (+ 187us) server_negotiation.cc:407] Connection header received
0812 14:52:22.188684 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.188685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.188695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.188746 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.189558 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.190038 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.191330 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.191538 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.191596 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.193946 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.193948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.193948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.193955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.195260 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.195620 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.195620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.195621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.195667 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.195843 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.195844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.195844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.195936 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.195980 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.196278 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.196350 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:52:22.196506 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.187959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:51965)
0812 14:52:22.188110 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.188150 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.188611 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.188796 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.188799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.189317 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.189490 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.189495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.190114 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.190115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.191259 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.191262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.191642 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.191643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.191777 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.192542 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.192553 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.193865 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.195342 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.195348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.195349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.195561 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.195711 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.195711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.195711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.195803 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.196004 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.196010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.196134 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.196223 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.196342 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260812 14:52:22.196935 16073 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46817
I20260812 14:52:22.197022 16073 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.197184 16073 heartbeater.cc:511] Master 127.4.27.62:46817 requested a full tablet report, sending...
I20260812 14:52:22.197538 15247 ts_manager.cc:194] Registered new tserver with Master: dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3:42207)
I20260812 14:52:22.197880 15247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:51965
I20260812 14:52:22.203171 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.27.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.61.keytab.
Entry for principal oryx/127.4.27.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.61.keytab.
I20260812 14:52:22.227375 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:36581
--webserver_interface=127.4.27.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.61.keytab
--principal=oryx/127.4.27.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46817,127.4.27.61:36581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:22.307205 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:22.307386 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:22.307407 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:22.308480 16083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:22.308511 16083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:22.308524 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:22.308535 16083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:22.308545 16083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:22.309932 16083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46817,127.4.27.61:36581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.61.keytab
--principal=oryx/127.4.27.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:36581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.27.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
W20260812 14:52:22.310026 16083 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.27.62:46817,127.4.27.61:36581'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 14:52:22.310129 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:22.310292 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:52:22.314837 16083 server_base.cc:1034] running on GCE node
W20260812 14:52:22.314785 16094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.314785 16096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.314775 16091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.315207 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.315477 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.316612 16083 hybrid_clock.cc:648] HybridClock initialized: now 1786546342316599 us; error 27 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.318866 16083 init.cc:377] Logged in from keytab as oryx/127.4.27.61@KRBTEST.COM (short username oryx)
I20260812 14:52:22.319283 16083 webserver.cc:460] Webserver started at http://127.4.27.61:37053/ using document root <none> and password file <none>
I20260812 14:52:22.319468 16083 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.319514 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.319634 16083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:22.320600 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/instance:
uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "6d455ec8909989ee1782c1818f9e96f0"
server_key_iv: "596fd5483b10efd6f34833667b22c443"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.320914 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal/instance:
uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "6d455ec8909989ee1782c1818f9e96f0"
server_key_iv: "596fd5483b10efd6f34833667b22c443"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.322263 16083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:52:22.322943 16108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.323087 16083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.323167 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "6d455ec8909989ee1782c1818f9e96f0"
server_key_iv: "596fd5483b10efd6f34833667b22c443"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.323231 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.339140 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.341094 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:22.341244 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.344844 16083 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.61:36581
I20260812 14:52:22.344873 16214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.61:36581 every 8 connection(s)
I20260812 14:52:22.345175 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
I20260812 14:52:22.346050 16220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:22.347478 16220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:52:22.352255 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 16083
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.61@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:22.355518 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.348074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.4.27.62:46817)
0812 14:52:22.348124 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.348125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.348298 (+ 173us) server_negotiation.cc:407] Connection header received
0812 14:52:22.348717 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.348718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.348727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.348780 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.349281 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.349757 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.350692 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.350866 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.350912 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.352787 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.352789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.352789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.352797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.354236 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.354607 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.354608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.354609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.354671 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.354915 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.354916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.354916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.355051 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.355097 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.355376 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.355436 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:52:22.355691 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.348035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:42448)
0812 14:52:22.348195 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.348233 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.348648 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.348828 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.348831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.349044 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.349219 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.349223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.349815 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.349815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.350631 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.350634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.350920 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.350920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.351010 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.351364 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.351371 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.352706 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.354315 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.354320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.354322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.354547 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.354725 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.354726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.354726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.354854 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.355103 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.355108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.355257 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.355355 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.355503 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260812 14:52:22.356329 16220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.61@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
I20260812 14:52:22.360893 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.352761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.4.27.61:36581)
0812 14:52:22.352901 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.352906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.352928 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:52:22.352949 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.352951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.352964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.353039 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.353639 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.354598 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.355472 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.355859 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.355926 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.358415 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.358424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.358424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.358431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.359786 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.360128 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.360131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.360131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.360194 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.360379 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.360379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.360379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.360534 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.360673 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.360734 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.360803 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260812 14:52:22.362974 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.356586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37036)
0812 14:52:22.356627 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.356636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.356667 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.356849 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.356850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.356979 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.357030 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.357032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.357711 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.357712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.358288 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.358291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.358778 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.358779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.358852 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.359146 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.359151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.360326 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.362234 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.362235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.362235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.362441 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.362583 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.362584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.362584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.362628 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.362784 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.362785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.362814 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.362856 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.362921 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:52:22.363068 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.356621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.4.27.61:36581)
0812 14:52:22.356739 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.356741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.356748 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:22.356766 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.356767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.356774 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.356828 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.357091 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.357622 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.358396 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.358707 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.358758 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.360402 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.360403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.360404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.360409 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.362171 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.362496 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.362496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.362497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.362542 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.362660 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.362661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.362661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.362746 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.362786 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.362872 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.362963 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260812 14:52:22.364104 16220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Bootstrap starting.
I20260812 14:52:22.364616 16220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:22.364831 16220 log.cc:824] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:22.365411 16220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: No bootstrap required, opened a new log
I20260812 14:52:22.366719 16220 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } }
I20260812 14:52:22.366827 16220 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:22.366847 16220 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bcc0c88f914d1eb38695e976d57ecd, State: Initialized, Role: FOLLOWER
I20260812 14:52:22.366928 16220 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } }
I20260812 14:52:22.367148 16220 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:22.367131 16234 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } } }
I20260812 14:52:22.367206 16234 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:52:22.368479 16249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:52:22.368530 16249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:52:22.369987 16083 master_runner.cc:428] Detected that this master d3bcc0c88f914d1eb38695e976d57ecd is joining an existing cluster
I20260812 14:52:22.370074 16083 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.27.61:36581
I20260812 14:52:22.376745 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.371223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.4.27.61:36581)
0812 14:52:22.371258 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.371260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.371315 (+ 55us) server_negotiation.cc:407] Connection header received
0812 14:52:22.371369 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.371370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.371377 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.371419 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.371714 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.372137 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.373232 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.373478 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.373551 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.374438 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.374439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.374440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.374446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.375698 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.376084 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.376085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.376085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.376156 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.376348 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.376349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.376349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.376454 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.376498 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.376630 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.376694 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:22.376902 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.371048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37040)
0812 14:52:22.371200 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.371258 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.371318 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.371424 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.371425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.371586 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.371653 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.371655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.372214 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.372215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.373104 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.373107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.373547 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.373549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.373619 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.373947 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.373951 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.374314 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.375774 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.375774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.375775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.375978 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.376226 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.376227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.376227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.376291 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.376505 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.376506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.376546 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.376615 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.376758 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":51,"thread_start_us":93,"threads_started":1}
I20260812 14:52:22.382287 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.371088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.4.27.62:46817)
0812 14:52:22.371142 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.371144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.377064 (+ 5920us) server_negotiation.cc:407] Connection header received
0812 14:52:22.377166 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.377167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.377179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.377235 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.377512 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.378177 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.379220 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.379409 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.379464 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.380255 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.380256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.380257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.380262 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.381410 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.381746 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.381746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.381746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.381785 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.381925 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.381925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.381925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.382005 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.382046 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.382170 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.382221 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:22.382367 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.371048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:42450)
0812 14:52:22.376976 (+ 5928us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.376985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.377088 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.377236 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.377237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.377378 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.377438 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.377440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.378244 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.378245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.379105 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.379108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.379474 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.379475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.379548 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.379849 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.379854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.380166 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.381479 (+ 1313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.381480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.381480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.381684 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.381821 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.381821 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.381822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.381875 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.382050 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.382051 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.382089 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.382154 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.382245 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5917,"mutex_wait_us":2,"spinlock_wait_cycles":8960,"thread_start_us":165,"threads_started":1}
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.62@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
I20260812 14:52:22.390169 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.384275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.4.27.61:36581)
0812 14:52:22.384312 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.384314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.384421 (+ 107us) server_negotiation.cc:407] Connection header received
0812 14:52:22.384492 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.384493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.384502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.384553 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.384921 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.385340 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.386035 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.386219 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.386277 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.388109 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.388110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.388111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.388116 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.389211 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.389607 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.389607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.389607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.389646 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.389794 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.389795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.389795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.389872 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.389912 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.390072 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.390120 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:22.390277 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.384221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37048)
0812 14:52:22.384328 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.384367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.384433 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.384559 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.384562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.384811 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.384853 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.384854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.385404 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.385405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.385964 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.385966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.386279 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.386280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.386376 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.386740 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.386752 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.388031 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.389280 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.389288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.389290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.389543 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.389684 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.389685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.389685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.389747 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.389920 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.389926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.389977 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.390058 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.390171 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260812 14:52:22.390795 15247 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.27.61:36581: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
member_type: NON_VOTER
last_known_addr {
host: "127.4.27.61"
port: 36581
}
attrs {
promote: true
}
}
dest_uuid: "129addd4a43b4f1b96b7b060cb647c6c"
cas_config_opid_index: -1
I20260812 14:52:22.391121 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } }
I20260812 14:52:22.391556 16265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d3bcc0c88f914d1eb38695e976d57ecd (127.4.27.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } }
I20260812 14:52:22.391844 16164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:22.392372 16265 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.27.61:36581
I20260812 14:52:22.392450 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } }
I20260812 14:52:22.392511 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:22.392699 16164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Refusing update from remote peer 129addd4a43b4f1b96b7b060cb647c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 14:52:22.392712 16083 master.cc:537] Master@127.4.27.61:36581 shutting down...
I20260812 14:52:22.394151 16264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:52:22.395110 15221 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 14:52:22.395215 16083 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:22.395306 16083 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:22.395357 16083 tablet_replica.cc:330] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: stopping tablet replica
W20260812 14:52:22.395639 15221 consensus_peers.cc:487] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c -> Peer d3bcc0c88f914d1eb38695e976d57ecd (127.4.27.61:36581): Couldn't send request to peer d3bcc0c88f914d1eb38695e976d57ecd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:52:22.408689 16083 master.cc:552] Master@127.4.27.61:36581 shutdown complete.
I20260812 14:52:22.408732 16083 master_runner.cc:305] Clearing existing system tablet
I20260812 14:52:22.408851 16083 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.408897 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.409580 16083 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.001s
I20260812 14:52:22.410130 16271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.410446 16083 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.410509 16083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:52:22.410534 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "6d455ec8909989ee1782c1818f9e96f0"
server_key_iv: "596fd5483b10efd6f34833667b22c443"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.411278 16083 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:22.412768 16083 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 14:52:22.412842 16083 log.cc:1196] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 14:52:22.413024 16083 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Deleting consensus metadata
I20260812 14:52:22.413142 16083 master_runner.cc:315] Copying system tablet from 127.4.27.62:46817
I20260812 14:52:22.413728 16083 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: tablet copy: Beginning tablet copy session from remote peer at address 127.4.27.62:46817
I20260812 14:52:22.419624 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.414100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.4.27.62:46817)
0812 14:52:22.414144 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.414146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.414207 (+ 61us) server_negotiation.cc:407] Connection header received
0812 14:52:22.414290 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.414291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.414298 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.414373 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.414695 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.415154 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.416230 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.416439 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.416494 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.417395 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.417396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.417397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.417402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.418657 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.419002 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.419003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.419003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.419045 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.419219 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.419220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.419220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.419322 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.419374 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.419513 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.419566 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:22.419693 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.414030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:42458)
0812 14:52:22.414137 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.414147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.414234 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.414373 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.414374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.414559 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.414633 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.414635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.415226 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.415227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.416100 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.416103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.416505 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.416506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.416572 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.416943 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.416948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.417321 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.418726 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.418727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.418728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.418940 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.419082 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.419083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.419083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.419141 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.419379 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.419380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.419424 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.419497 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.419596 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":66,"threads_started":1}
I20260812 14:52:22.420284 15286 tablet_copy_service.cc:140] P 129addd4a43b4f1b96b7b060cb647c6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d3bcc0c88f914d1eb38695e976d57ecd ({username='oryx', principal='oryx/127.4.27.61@KRBTEST.COM'} at 127.0.0.1:42458)
I20260812 14:52:22.420423 15286 tablet_copy_service.cc:161] P 129addd4a43b4f1b96b7b060cb647c6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d3bcc0c88f914d1eb38695e976d57ecd at {username='oryx', principal='oryx/127.4.27.61@KRBTEST.COM'} at 127.0.0.1:42458: session id = d3bcc0c88f914d1eb38695e976d57ecd-00000000000000000000000000000000
I20260812 14:52:22.421381 15286 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:52:22.421962 16083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 14:52:22.423209 16083 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: tablet copy: Starting download of 0 data blocks...
I20260812 14:52:22.423363 16083 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: tablet copy: Starting download of 1 WAL segments...
I20260812 14:52:22.424240 16083 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:52:22.425056 15286 tablet_copy_service.cc:342] P 129addd4a43b4f1b96b7b060cb647c6c: Request end of tablet copy session d3bcc0c88f914d1eb38695e976d57ecd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.27.61@KRBTEST.COM'} at 127.0.0.1:42458
I20260812 14:52:22.425110 15286 tablet_copy_service.cc:433] P 129addd4a43b4f1b96b7b060cb647c6c: ending tablet copy session d3bcc0c88f914d1eb38695e976d57ecd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d3bcc0c88f914d1eb38695e976d57ecd
I20260812 14:52:22.427430 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.428462 16285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.428625 16288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.428814 16083 server_base.cc:1034] running on GCE node
W20260812 14:52:22.430452 16286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.430568 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.430692 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.431804 16083 hybrid_clock.cc:648] HybridClock initialized: now 1786546342431796 us; error 23 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.432906 16083 init.cc:377] Logged in from keytab as oryx/127.4.27.61@KRBTEST.COM (short username oryx)
I20260812 14:52:22.433130 16083 webserver.cc:460] Webserver started at http://127.4.27.61:36947/ using document root <none> and password file <none>
I20260812 14:52:22.433235 16083 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.433279 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.433781 16083 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:22.434240 16296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.434391 16083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.434435 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "6d455ec8909989ee1782c1818f9e96f0"
server_key_iv: "596fd5483b10efd6f34833667b22c443"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.434486 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.448143 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.449592 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.452425 16083 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.61:36581
I20260812 14:52:22.452445 16392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.61:36581 every 8 connection(s)
I20260812 14:52:22.452720 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
I20260812 14:52:22.453729 16397 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:52:22.454376 16397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Bootstrap starting.
I20260812 14:52:22.457434 16397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:52:22.457653 16397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Bootstrap complete.
I20260812 14:52:22.457870 16397 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } }
I20260812 14:52:22.457963 16397 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d3bcc0c88f914d1eb38695e976d57ecd, State: Initialized, Role: LEARNER
I20260812 14:52:22.458009 16397 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } }
I20260812 14:52:22.458173 16397 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:22.458171 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } }
I20260812 14:52:22.458266 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: LEARNER
I20260812 14:52:22.459619 16422 catalog_manager.cc:1259] Loaded cluster ID: 16d04f3c2f684628800b7ec91f2a8a6d
I20260812 14:52:22.459664 16422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: loading cluster ID for follower catalog manager: success
I20260812 14:52:22.460868 16422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: acquiring CA information for follower catalog manager: success
I20260812 14:52:22.461131 16422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:52:22.467916 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.462180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.4.27.61:36581)
0812 14:52:22.462281 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.462282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.462289 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:22.462305 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.462306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.462314 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.462386 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.462876 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.463503 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.464189 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.464385 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.464440 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.465671 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.465673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.465673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.465680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.466959 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.467245 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.467247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.467247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.467298 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.467454 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.467455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.467455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.467553 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.467641 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.467734 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.467818 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.4.27.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.60.keytab.
Entry for principal oryx/127.4.27.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.60.keytab.
I20260812 14:52:22.494092 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:44601
--webserver_interface=127.4.27.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.60.keytab
--principal=oryx/127.4.27.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:22.575256 16427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:22.575439 16427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:22.575459 16427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:22.576571 16427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:22.576609 16427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:22.576622 16427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:22.576633 16427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:22.576644 16427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:22.578148 16427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.60.keytab
--principal=oryx/127.4.27.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.27.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:22.578359 16427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:22.578562 16427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.583014 16438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.583022 16436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.583031 16443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.583302 16427 server_base.cc:1034] running on GCE node
I20260812 14:52:22.583493 16427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.583729 16427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.584853 16427 hybrid_clock.cc:648] HybridClock initialized: now 1786546342584844 us; error 25 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.587129 16427 init.cc:377] Logged in from keytab as oryx/127.4.27.60@KRBTEST.COM (short username oryx)
I20260812 14:52:22.587522 16427 webserver.cc:460] Webserver started at http://127.4.27.60:35821/ using document root <none> and password file <none>
I20260812 14:52:22.587708 16427 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.587755 16427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.587877 16427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:22.588835 16427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/instance:
uuid: "99b73efbee8b472195586a150bca03cb"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "697a17115ed51e69e2f79f89c5eed0e3"
server_key_iv: "5709aa3b26b15266b829eae293dfc634"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.589149 16427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal/instance:
uuid: "99b73efbee8b472195586a150bca03cb"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "697a17115ed51e69e2f79f89c5eed0e3"
server_key_iv: "5709aa3b26b15266b829eae293dfc634"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.590327 16427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:22.590971 16459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.591141 16427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:22.591234 16427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
uuid: "99b73efbee8b472195586a150bca03cb"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "697a17115ed51e69e2f79f89c5eed0e3"
server_key_iv: "5709aa3b26b15266b829eae293dfc634"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.591295 16427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.601240 16427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.603155 16427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:22.603312 16427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.607757 16427 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.60:44601
I20260812 14:52:22.607803 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.60:44601 every 8 connection(s)
I20260812 14:52:22.608165 16427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
I20260812 14:52:22.609112 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 16427
I20260812 14:52:22.609934 16549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:22.611788 16549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.27.60@KRBTEST.COM
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.60@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:22.619319 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.609745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.4.27.60:44601)
0812 14:52:22.609926 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.609935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.609966 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:52:22.610498 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.610506 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.610569 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.610707 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.611231 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.612455 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.613321 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.613701 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.613801 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.616182 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.616189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.616189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.616196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.617754 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.618149 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.618152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.618152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.618233 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.618478 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.618479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.618480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.618779 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.618883 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.619003 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.619119 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260812 14:52:22.619560 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.612380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:42462)
0812 14:52:22.612568 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.612593 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.612647 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.612768 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.612771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.613024 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.613089 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.613091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.613759 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.613760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.614559 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.614562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.615021 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.615022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.615070 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.615474 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.615483 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.616841 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.618476 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.618482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.618484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.618737 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.618923 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.618924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.618924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.618987 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.619228 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.619233 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.619292 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.619375 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.619474 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260812 14:52:22.619647 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.612552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.4.27.62:46817)
0812 14:52:22.612599 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.612601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.612639 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:52:22.612688 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.612689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.612698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.612753 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.613159 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.613679 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.614645 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.614948 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.615012 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.616946 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.616948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.616948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.616956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.618409 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.618805 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.618806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.618806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.618864 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.619051 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.619052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.619052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.619179 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.619230 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.619498 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.619566 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:22.620157 16549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.60@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
I20260812 14:52:22.625303 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.620361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37078)
0812 14:52:22.620401 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.620410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.620443 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.620566 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.620567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.620718 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.620764 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.620766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.621379 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.621380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.621908 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.621910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.622204 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.622205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.622249 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.622548 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.622551 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.623511 (+ 960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.624647 (+ 1136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.624648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.624648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.624792 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.624923 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.624924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.624924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.624964 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.625109 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.625109 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.625137 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.625194 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.625252 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:52:22.625329 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.620435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.4.27.61:36581)
0812 14:52:22.620473 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.620475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.620482 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:22.620508 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.620508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.620521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.620565 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.620813 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.621324 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.621972 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.622157 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.622199 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.623590 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.623591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.623591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.623598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.624587 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.624845 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.624845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.624846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.624884 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.624994 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.624994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.624995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.625072 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.625113 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.625211 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.625273 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:52:22.625645 16549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.60@KRBTEST.COM for oryx/127.4.27.60@KRBTEST.COM
I20260812 14:52:22.630788 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.625842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.0.0.1:47928)
0812 14:52:22.625881 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.625890 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.625923 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.626029 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.626030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.626142 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.626164 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.626165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.626801 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.626802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.627270 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.627271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.627579 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.627580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.627623 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.627947 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.627951 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.628893 (+ 942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.629966 (+ 1073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.629966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.629967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.630106 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.630261 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.630261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.630262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.630312 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.630522 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.630523 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.630556 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.630614 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.630702 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:52:22.630849 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.625890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.4.27.60:44601)
0812 14:52:22.625920 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.625921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.625928 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:22.625973 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.625973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.625981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.626018 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.626219 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.626736 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.627332 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.627526 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.627576 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.628960 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.628961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.628961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.628967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.629901 (+ 934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.630161 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.630162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.630162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.630207 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.630377 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.630378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.630378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.630472 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.630522 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.630640 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.630755 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":17}
I20260812 14:52:22.631925 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Bootstrap starting.
I20260812 14:52:22.632651 16549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:22.632867 16549 log.cc:824] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:22.633476 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: No bootstrap required, opened a new log
I20260812 14:52:22.634871 16549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } }
I20260812 14:52:22.634984 16549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:22.635006 16549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b73efbee8b472195586a150bca03cb, State: Initialized, Role: FOLLOWER
I20260812 14:52:22.635087 16549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } }
I20260812 14:52:22.635318 16549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:22.635273 16559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } } }
I20260812 14:52:22.635371 16559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:52:22.636933 16577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:52:22.636988 16577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:52:22.638943 16427 master_runner.cc:428] Detected that this master 99b73efbee8b472195586a150bca03cb is joining an existing cluster
I20260812 14:52:22.638983 16427 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.27.60:44601
I20260812 14:52:22.645941 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.640189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.0.0.1:47938)
0812 14:52:22.640350 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.640359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.640419 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.640547 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.640548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.640699 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.640747 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.640749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.641534 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.641534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.642396 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.642399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.642858 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.642911 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.643292 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.643297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.643586 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.645133 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.645134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.645134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.645286 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.645473 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.645474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.645474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.645517 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.645737 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.645738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.645766 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.645808 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.645876 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260812 14:52:22.646032 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.640128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.4.27.60:44601)
0812 14:52:22.640157 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.640159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.640402 (+ 243us) server_negotiation.cc:407] Connection header received
0812 14:52:22.640473 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.640473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.640480 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.640502 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.640815 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.641464 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642480 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.642750 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642785 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.643662 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.643663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.643663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.643670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.645050 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.645349 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.645350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.645350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.645412 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.645565 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.645565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.645566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.645680 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.645756 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.645831 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.645935 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":15}
I20260812 14:52:22.665583 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.639806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:42466)
0812 14:52:22.639950 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.639959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.640029 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.640191 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.640191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.640465 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.640552 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.640555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.641456 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.641457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.642474 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.642477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.642973 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.643039 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.662843 (+ 19804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.662848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.663255 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.664621 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.664622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.664622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.664815 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.664978 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.664978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.664979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.665033 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.665284 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.665285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.665340 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.665400 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.665488 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":65,"thread_start_us":89,"threads_started":1}
I20260812 14:52:22.665721 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.639912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37086)
0812 14:52:22.640021 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.640030 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.640111 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.640281 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.640281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.640512 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.640571 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.640573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.641467 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.641468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.642416 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.642418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.642869 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.642923 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.662683 (+ 19760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.662688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.663037 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.664684 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.664685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.664685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.664926 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.665121 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.665121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.665122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.665169 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.665410 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.665411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.665455 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.665525 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.665618 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":77,"thread_start_us":76,"threads_started":1}
I20260812 14:52:22.665741 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.639859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.4.27.61:36581)
0812 14:52:22.639900 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.639901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.640077 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:52:22.640186 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.640186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.640195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.640222 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.640652 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.641384 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642494 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.642773 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642807 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.663116 (+ 20309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.663117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.663118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.663125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.664581 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.665019 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.665020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.665020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.665073 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.665226 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.665226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.665227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.665341 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.665407 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.665557 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.665642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:52:22.666270 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.639837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.4.27.62:46817)
0812 14:52:22.639876 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.639878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.640007 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:52:22.640097 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.640098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.640108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.640135 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.640642 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.641366 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642564 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.642863 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.642923 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.663347 (+ 20424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.663349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.663349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.663356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.664546 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.664874 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.664875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.664875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.664927 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.665089 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.665090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.665091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.665225 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.665285 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.666108 (+ 823us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.666193 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:52:22.670553 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 15363
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.62@KRBTEST.COM for oryx/127.4.27.60@KRBTEST.COM
I20260812 14:52:22.685496 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.668042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.4.27.60:44601)
0812 14:52:22.668080 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.668081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.668120 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:52:22.668174 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.668175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.668184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.668240 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.668446 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.669142 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.669738 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.670060 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.670124 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.671861 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.671861 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.671862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.671867 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.684353 (+ 12486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.684727 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.684728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.684729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.684794 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.684972 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.684972 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.684973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.685087 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.685160 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.685346 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.685421 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:22.685504 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.668004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.0.0.1:47946)
0812 14:52:22.668083 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.668091 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.668130 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.668243 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.668243 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.668360 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.668394 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.668395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.669205 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.669205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.669673 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.669674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.670127 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.670128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.670203 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.670632 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.670637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.671786 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.684452 (+ 12666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.684453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.684453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.684652 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.684857 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.684858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.684859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.684913 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.685189 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.685190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.685236 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.685303 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.685405 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 14:52:22.686435 15247 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.27.60:44601: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "99b73efbee8b472195586a150bca03cb"
member_type: NON_VOTER
last_known_addr {
host: "127.4.27.60"
port: 44601
}
attrs {
promote: true
}
}
dest_uuid: "129addd4a43b4f1b96b7b060cb647c6c"
cas_config_opid_index: 5
I20260812 14:52:22.686697 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } }
I20260812 14:52:22.687052 16265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 99b73efbee8b472195586a150bca03cb (127.4.27.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.688030 16265 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.27.60:44601
I20260812 14:52:22.688097 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.688155 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:22.688489 16491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:22.689378 16491 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Refusing update from remote peer 129addd4a43b4f1b96b7b060cb647c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 14:52:22.689694 16427 master.cc:537] Master@127.4.27.60:44601 shutting down...
I20260812 14:52:22.689744 16264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:52:22.690464 15221 consensus_peers.cc:487] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c -> Peer 99b73efbee8b472195586a150bca03cb (127.4.27.60:44601): Couldn't send request to peer 99b73efbee8b472195586a150bca03cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:52:22.691012 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:33559
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:46817
--webserver_port=41171
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:52:22.693112 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.687122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37102)
0812 14:52:22.687242 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.687251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.687289 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.687975 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.687976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.687978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:22.688016 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.688017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.688918 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.688919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.690095 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.690098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.692855 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.692856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.692926 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.692950 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.693009 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.693046 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 14:52:22.693127 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.687734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.4.27.61:36581)
0812 14:52:22.687801 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.687803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.687810 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:22.687833 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.687834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.687857 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.687916 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:22.688082 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.688827 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.690188 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.691284 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.691351 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.691422 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.693018 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.693063 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:52:22.693400 16353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 LEARNER]: Refusing update from remote peer 129addd4a43b4f1b96b7b060cb647c6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 14:52:22.693698 16265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 14:52:22.694242 16353 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 99b73efbee8b472195586a150bca03cb (127.4.27.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.695565 16265 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: attempting to promote NON_VOTER d3bcc0c88f914d1eb38695e976d57ecd to VOTER
I20260812 14:52:22.695566 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129addd4a43b4f1b96b7b060cb647c6c. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.695649 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: LEARNER
I20260812 14:52:22.695684 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: NON_VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: true } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.695719 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: LEARNER
I20260812 14:52:22.695777 16265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } }
W20260812 14:52:22.696548 15221 consensus_peers.cc:487] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c -> Peer 99b73efbee8b472195586a150bca03cb (127.4.27.60:44601): Couldn't send request to peer 99b73efbee8b472195586a150bca03cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 14:52:22.696658 16353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 LEARNER]: Refusing update from remote peer 129addd4a43b4f1b96b7b060cb647c6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 14:52:22.696842 16264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 14:52:22.697316 16427 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:22.697407 16427 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:22.697456 16427 tablet_replica.cc:330] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: stopping tablet replica
I20260812 14:52:22.698477 16600 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3bcc0c88f914d1eb38695e976d57ecd (127.4.27.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.698611 16353 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d3bcc0c88f914d1eb38695e976d57ecd (127.4.27.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.699420 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.699503 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:52:22.699492 16600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.699685 16600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:22.699726 16264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.699831 16264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:22.711961 16427 master.cc:552] Master@127.4.27.60:44601 shutdown complete.
I20260812 14:52:22.712011 16427 master_runner.cc:305] Clearing existing system tablet
I20260812 14:52:22.712131 16427 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.712173 16427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.712849 16427 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:22.713352 16606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.714462 16427 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.714763 16427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260812 14:52:22.714833 16427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
uuid: "99b73efbee8b472195586a150bca03cb"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "697a17115ed51e69e2f79f89c5eed0e3"
server_key_iv: "5709aa3b26b15266b829eae293dfc634"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.715553 16427 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:22.716889 16427 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 14:52:22.716958 16427 log.cc:1196] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 14:52:22.717133 16427 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Deleting consensus metadata
I20260812 14:52:22.717260 16427 master_runner.cc:315] Copying system tablet from 127.4.27.62:46817
I20260812 14:52:22.717865 16427 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: tablet copy: Beginning tablet copy session from remote peer at address 127.4.27.62:46817
I20260812 14:52:22.724885 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.718227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.4.27.62:46817)
0812 14:52:22.718270 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.718272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.718330 (+ 58us) server_negotiation.cc:407] Connection header received
0812 14:52:22.718422 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.718423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.718433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.718498 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.718863 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.719600 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.720595 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.720907 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.720965 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.722166 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.722168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.722168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.722175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.723746 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.724135 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.724136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.724136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.724196 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.724393 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.724394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.724395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.724521 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.724583 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.724729 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.724803 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:22.724903 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.718175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:42474)
0812 14:52:22.718272 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.718282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.718369 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.718505 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.718506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.718707 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.718787 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.718789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.719691 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.719692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.720504 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.720507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.720967 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.720968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.721040 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.721555 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.721560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.722050 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.723853 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.723854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.723855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.724060 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.724264 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.724264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.724265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.724333 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.724576 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.724577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.724626 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.724701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.724811 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
I20260812 14:52:22.725409 15286 tablet_copy_service.cc:140] P 129addd4a43b4f1b96b7b060cb647c6c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 99b73efbee8b472195586a150bca03cb ({username='oryx', principal='oryx/127.4.27.60@KRBTEST.COM'} at 127.0.0.1:42474)
I20260812 14:52:22.725461 15286 tablet_copy_service.cc:161] P 129addd4a43b4f1b96b7b060cb647c6c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 99b73efbee8b472195586a150bca03cb at {username='oryx', principal='oryx/127.4.27.60@KRBTEST.COM'} at 127.0.0.1:42474: session id = 99b73efbee8b472195586a150bca03cb-00000000000000000000000000000000
I20260812 14:52:22.725988 15286 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 14:52:22.726545 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 14:52:22.727706 16427 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: tablet copy: Starting download of 0 data blocks...
I20260812 14:52:22.727903 16427 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: tablet copy: Starting download of 1 WAL segments...
I20260812 14:52:22.728677 16427 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 14:52:22.729467 15286 tablet_copy_service.cc:342] P 129addd4a43b4f1b96b7b060cb647c6c: Request end of tablet copy session 99b73efbee8b472195586a150bca03cb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.27.60@KRBTEST.COM'} at 127.0.0.1:42474
I20260812 14:52:22.729512 15286 tablet_copy_service.cc:433] P 129addd4a43b4f1b96b7b060cb647c6c: ending tablet copy session 99b73efbee8b472195586a150bca03cb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 99b73efbee8b472195586a150bca03cb
I20260812 14:52:22.732416 16427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.733464 16617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.735260 16618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.735630 16427 server_base.cc:1034] running on GCE node
W20260812 14:52:22.735755 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.735972 16427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.736169 16427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.737330 16427 hybrid_clock.cc:648] HybridClock initialized: now 1786546342737321 us; error 25 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.738682 16427 init.cc:377] Logged in from keytab as oryx/127.4.27.60@KRBTEST.COM (short username oryx)
I20260812 14:52:22.738920 16427 webserver.cc:460] Webserver started at http://127.4.27.60:32797/ using document root <none> and password file <none>
I20260812 14:52:22.739024 16427 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.739056 16427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.739683 16427 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.740190 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.740327 16427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.740372 16427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
uuid: "99b73efbee8b472195586a150bca03cb"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "697a17115ed51e69e2f79f89c5eed0e3"
server_key_iv: "5709aa3b26b15266b829eae293dfc634"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.740417 16427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.758569 16427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.760404 16427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.763456 16427 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.60:44601
I20260812 14:52:22.763541 16745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.60:44601 every 8 connection(s)
I20260812 14:52:22.763783 16427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
I20260812 14:52:22.764628 16746 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:52:22.765137 16746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Bootstrap starting.
I20260812 14:52:22.767969 16746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:52:22.768177 16746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: Bootstrap complete.
I20260812 14:52:22.768368 16746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } }
I20260812 14:52:22.768469 16746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99b73efbee8b472195586a150bca03cb, State: Initialized, Role: LEARNER
I20260812 14:52:22.768510 16746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } }
I20260812 14:52:22.769059 16746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:22.769286 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:22.769469 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: This master's current role is: LEARNER
I20260812 14:52:22.770625 16767 catalog_manager.cc:1259] Loaded cluster ID: 16d04f3c2f684628800b7ec91f2a8a6d
I20260812 14:52:22.770715 16767 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: loading cluster ID for follower catalog manager: success
I20260812 14:52:22.771971 16767 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: acquiring CA information for follower catalog manager: success
I20260812 14:52:22.772262 16767 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 14:52:22.786103 16596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:22.786283 16596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:22.786305 16596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:22.787420 16596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:22.787458 16596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:22.787487 16596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:22.788937 16596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:33559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=41171
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:22.789153 16596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:22.789364 16596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.794564 16776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.794575 16781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.794581 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.794606 16596 server_base.cc:1034] running on GCE node
I20260812 14:52:22.794958 16596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.795184 16596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.796326 16596 hybrid_clock.cc:648] HybridClock initialized: now 1786546342796310 us; error 29 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.798676 16596 init.cc:377] Logged in from keytab as oryx/127.4.27.1@KRBTEST.COM (short username oryx)
I20260812 14:52:22.799086 16596 webserver.cc:460] Webserver started at http://127.4.27.1:41171/ using document root <none> and password file <none>
I20260812 14:52:22.799276 16596 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.799340 16596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.800568 16596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:22.801095 16798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.801239 16596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.801319 16596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "3db8e50d4dcf4369bc5473ee39163e7a"
format_stamp: "Formatted at 2026-08-12 14:52:21 on dist-test-slave-6zbq"
server_key: "15b236dac4864d0e85f714624545c40e"
server_key_iv: "2b328e7306f1923ac8f66c862fe2caef"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.801620 16596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.811188 16596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.812711 16596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:22.812845 16596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.813024 16596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:22.813285 16596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:22.813318 16596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.813344 16596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:22.813361 16596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.818893 16596 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:33559
I20260812 14:52:22.818948 17007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:33559 every 8 connection(s)
I20260812 14:52:22.819248 16596 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.60@KRBTEST.COM
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
I20260812 14:52:22.829341 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 16596
I20260812 14:52:22.829463 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.819877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:36921 (local address 127.4.27.62:46817)
0812 14:52:22.819924 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.819926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.820172 (+ 246us) server_negotiation.cc:407] Connection header received
0812 14:52:22.821208 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.821209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.821219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.821247 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.822038 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.822808 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.823985 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.824234 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.824293 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.826364 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.826366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.826366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.826373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.828181 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.828532 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.828533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.828533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.828589 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.828830 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.828831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.828831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.828934 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.828988 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.829298 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.829375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:22.829505 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 15593
I20260812 14:52:22.830046 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.819883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:36921)
0812 14:52:22.820052 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.820088 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.821069 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.821397 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.821399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.821734 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.821925 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.821931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.822869 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.822870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.823913 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.823915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.824303 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.824304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.824367 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.824845 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.824850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.826248 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.828252 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.828259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.828261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.828467 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.828641 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.828641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.828642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.828745 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.828990 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.828997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.829132 (+ 135us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.829719 (+ 587us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.829866 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":18,"thread_start_us":105,"threads_started":1}
I20260812 14:52:22.830091 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.819886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:57913 (local address 127.4.27.60:44601)
0812 14:52:22.819995 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.819996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.820180 (+ 184us) server_negotiation.cc:407] Connection header received
0812 14:52:22.821192 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.821193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.821209 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.821254 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.822009 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.823047 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.823806 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.824055 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.824117 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.826819 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.826820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.826821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.826828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.828545 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.829270 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.829271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.829272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.829338 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.829533 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.829533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.829534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.829654 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.829693 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.829895 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.829984 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260812 14:52:22.830412 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.819883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.4.27.1:57913)
0812 14:52:22.820049 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.820088 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.821061 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.821396 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.821399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.821719 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.821924 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.821929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.823120 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.823121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.823735 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.823737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.824134 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.824135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.824281 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.824731 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.824739 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.826727 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.828779 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.828780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.828780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.829187 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.829411 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.829411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.829412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.829466 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.829734 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.829735 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.829783 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.830167 (+ 384us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.830269 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260812 14:52:22.836588 17013 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46817
I20260812 14:52:22.836694 17013 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.836796 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.820341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:45031 (local address 127.4.27.61:36581)
0812 14:52:22.820395 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.820397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.820405 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:22.821200 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.821201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.821212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.821244 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.822031 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.822641 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.823998 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.824239 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.824301 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.827281 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.827282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.827282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.827290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.828705 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.829011 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.829012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.829012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.829067 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.829271 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.829272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.829272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.829419 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.829461 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.836629 (+ 7168us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.836713 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:52:22.836867 17013 heartbeater.cc:511] Master 127.4.27.62:46817 requested a full tablet report, sending...
I20260812 14:52:22.837316 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.820160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.4.27.1:45031)
0812 14:52:22.820282 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.820292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.821074 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.821397 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.821399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.821738 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.821925 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.821931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.822752 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.822753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.823889 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.823892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.824355 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.824356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.824415 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.824971 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.824975 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.827211 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.828767 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.828768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.828768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.828944 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.829125 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.829125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.829126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.829184 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.829510 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.836452 (+ 6942us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.836502 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.837075 (+ 573us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.837211 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":25,"thread_start_us":71,"threads_started":1}
I20260812 14:52:22.837297 15247 ts_manager.cc:194] Re-registered known tserver with Master: 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559)
I20260812 14:52:22.837724 15247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:36921
I20260812 14:52:22.838608 17016 heartbeater.cc:347] Connected to a master server at 127.4.27.61:36581
I20260812 14:52:22.838658 17016 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.838721 17016 heartbeater.cc:511] Master 127.4.27.61:36581 requested a full tablet report, sending...
I20260812 14:52:22.839049 16335 ts_manager.cc:194] Registered new tserver with Master: 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559)
I20260812 14:52:22.842821 17012 heartbeater.cc:347] Connected to a master server at 127.4.27.60:44601
I20260812 14:52:22.842869 17012 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.842907 17012 heartbeater.cc:511] Master 127.4.27.60:44601 requested a full tablet report, sending...
I20260812 14:52:22.843195 16671 ts_manager.cc:194] Registered new tserver with Master: 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559)
I20260812 14:52:22.846932 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:36901
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:46817
--webserver_port=43271
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:22.926047 17030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:22.926237 17030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:22.926259 17030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:22.927374 17030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:22.927414 17030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:22.927439 17030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:22.928826 17030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:36901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=43271
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:22.928996 17030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:22.929164 17030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:22.933885 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:22.933921 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.934063 17030 server_base.cc:1034] running on GCE node
W20260812 14:52:22.933893 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:22.934383 17030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:22.934592 17030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:22.935731 17030 hybrid_clock.cc:648] HybridClock initialized: now 1786546342935717 us; error 24 us; skew 500 ppm
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:22.938027 17030 init.cc:377] Logged in from keytab as oryx/127.4.27.2@KRBTEST.COM (short username oryx)
I20260812 14:52:22.938534 17030 webserver.cc:460] Webserver started at http://127.4.27.2:43271/ using document root <none> and password file <none>
I20260812 14:52:22.938725 17030 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:22.938786 17030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:22.940057 17030 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:22.940671 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.940810 17030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:22.940891 17030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "b7a80beb19f348b8a7835bf593fa8c2b"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "bc477a496085e48198428f0b74336916"
server_key_iv: "b6e53ccb169e0c427da7ddc24ebbdd47"
server_key_version: "encryptionkey@0"
I20260812 14:52:22.941224 17030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:22.951153 17030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:22.952723 17030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:22.952845 17030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:22.953027 17030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:22.953297 17030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:22.953330 17030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.953356 17030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:22.953374 17030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:22.959165 17030 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36901
I20260812 14:52:22.959239 17250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36901 every 8 connection(s)
I20260812 14:52:22.959517 17030 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:22.961313 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 17030
I20260812 14:52:22.961467 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 15841
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
I20260812 14:52:22.976380 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.960479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:40509 (local address 127.4.27.61:36581)
0812 14:52:22.960527 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.960529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.967033 (+ 6504us) server_negotiation.cc:407] Connection header received
0812 14:52:22.967147 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.967148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.967159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.967184 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.967956 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.968563 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.969902 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.970151 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.970187 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.973066 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.973068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.973068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.973076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.974659 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.975131 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.975132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.975132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.975194 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.975464 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.975465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.975465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.975769 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.975837 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.976200 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.976287 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:22.976925 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.960415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.4.27.2:40509)
0812 14:52:22.961484 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.961493 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.967064 (+ 5571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.967325 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.967329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.967634 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.967875 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.967879 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.968655 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.968655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.969788 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.969791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.970274 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.970275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.970425 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.971500 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.971510 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.972968 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.974745 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.974752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.974754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.975036 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.975262 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.975263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.975263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.975395 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.975881 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.975887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.976038 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.976509 (+ 471us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.976697 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":66,"threads_started":1}
I20260812 14:52:22.977509 17259 heartbeater.cc:347] Connected to a master server at 127.4.27.61:36581
I20260812 14:52:22.977599 17259 heartbeater.cc:464] Registering TS with master...
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:22.977784 17259 heartbeater.cc:511] Master 127.4.27.61:36581 requested a full tablet report, sending...
I20260812 14:52:22.978210 16335 ts_manager.cc:194] Registered new tserver with Master: b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:22.980520 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.960149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:52471 (local address 127.4.27.62:46817)
0812 14:52:22.960538 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.960540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.960552 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:52:22.961419 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.961421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.961434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.961485 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.974093 (+ 12608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.974869 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.975831 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.976101 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.976140 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.978182 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.978183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.978184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.978191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.979530 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.979840 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.979841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.979842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.979899 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.980059 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.980059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.980060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.980172 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.980230 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.980362 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.980433 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260812 14:52:22.980540 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.960134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:52471)
0812 14:52:22.960301 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.960341 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.961318 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.973541 (+ 12223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.973543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.973709 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.973777 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.973779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.974977 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.974977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.975735 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.975737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.976220 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.976220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.976287 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.976727 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.976732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.978089 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.979618 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.979619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.979619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.979778 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.979951 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.979952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.979952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.980009 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.980226 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.980227 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.980271 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.980333 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.980430 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260812 14:52:22.980831 17257 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46817
I20260812 14:52:22.980873 17257 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.980927 17257 heartbeater.cc:511] Master 127.4.27.62:46817 requested a full tablet report, sending...
I20260812 14:52:22.981184 15247 ts_manager.cc:194] Re-registered known tserver with Master: b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:22.981496 15247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:52471
Aug 12 14:52:22 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.60@KRBTEST.COM
I20260812 14:52:22.996227 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:42207
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:46817
--webserver_port=38417
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:52:22.998126 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.960187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:38565 (local address 127.4.27.60:44601)
0812 14:52:22.960241 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:22.960250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:22.960591 (+ 341us) server_negotiation.cc:407] Connection header received
0812 14:52:22.961505 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:22.961506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:22.961524 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:22.961555 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:22.992057 (+ 30502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.992543 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.993507 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.993693 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.993735 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.995598 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:22.995600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:22.995600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:22.995608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:22.997103 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.997413 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.997414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.997414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.997476 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:22.997654 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:22.997655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:22.997655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:22.997786 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:22.997835 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:22.997993 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 14:52:22.998062 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:52:22.998200 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:22.960135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.4.27.2:38565)
0812 14:52:22.960534 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:22.960544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:22.961330 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:22.991697 (+ 30367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:22.991699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:22.991894 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:22.991959 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.991961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.992639 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.992640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.993431 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:22.993434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:22.993758 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:22.993759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:22.993832 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:22.994230 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:22.994235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:22.995512 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:22.997195 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.997195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.997196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.997353 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.997535 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:22.997535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:22.997536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:22.997597 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:22.997837 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:22.997838 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:22.997893 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 14:52:22.997973 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 14:52:22.998073 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":100,"threads_started":1}
I20260812 14:52:22.998605 17251 heartbeater.cc:347] Connected to a master server at 127.4.27.60:44601
I20260812 14:52:22.998663 17251 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:22.998750 17251 heartbeater.cc:511] Master 127.4.27.60:44601 requested a full tablet report, sending...
I20260812 14:52:22.999058 16671 ts_manager.cc:194] Registered new tserver with Master: b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
W20260812 14:52:23.076937 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:23.077118 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:23.077139 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:23.078236 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:23.078279 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:23.078305 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:23.079749 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:42207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=38417
--tserver_master_addrs=127.4.27.62:46817,127.4.27.61:36581,127.4.27.60:44601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:23.079942 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:23.080118 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:23.084685 17279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:23.084806 17268 server_base.cc:1034] running on GCE node
W20260812 14:52:23.084884 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:23.084697 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:23.085103 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:23.085318 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:23.086452 17268 hybrid_clock.cc:648] HybridClock initialized: now 1786546343086437 us; error 30 us; skew 500 ppm
Aug 12 14:52:23 dist-test-slave-6zbq krb5kdc[15164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546343, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:23.088701 17268 init.cc:377] Logged in from keytab as oryx/127.4.27.3@KRBTEST.COM (short username oryx)
I20260812 14:52:23.089059 17268 webserver.cc:460] Webserver started at http://127.4.27.3:38417/ using document root <none> and password file <none>
I20260812 14:52:23.089243 17268 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:23.089298 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:23.090570 17268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:23.091127 17290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:23.091292 17268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:23.091364 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "dfd3cd8bc4464b4088952aef04a0d5df"
format_stamp: "Formatted at 2026-08-12 14:52:22 on dist-test-slave-6zbq"
server_key: "d18492160bd9ae3ea11670d4eceaa602"
server_key_iv: "ef5714f0f26eac4a0a4e3abd0b50d3ff"
server_key_version: "encryptionkey@0"
I20260812 14:52:23.091682 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:23.111013 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:23.113232 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:23.113363 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:23.113610 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:23.113952 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:23.113996 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:23.114027 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:23.114043 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:23.119482 17268 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42207
I20260812 14:52:23.119536 17504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42207 every 8 connection(s)
I20260812 14:52:23.119860 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:23.121491 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 17268
I20260812 14:52:23.125813 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.121472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.0.0.1:47950)
0812 14:52:23.122321 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.122333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.122391 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.122691 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.122691 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.122693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:23.122739 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.122742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123709 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.123710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.124830 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.124834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125602 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125657 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.125660 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.125687 (+ 27us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.125731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831}
I20260812 14:52:23.126180 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.121528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.4.27.60:44601)
0812 14:52:23.122548 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.122550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.122556 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:52:23.122571 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.122571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.122593 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.122815 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:23.122851 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123613 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.124932 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125518 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125980 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.126052 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.126072 (+ 20us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.126109 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":65,"threads_started":1}
I20260812 14:52:23.126441 16688 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 14:52:23.126647 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129addd4a43b4f1b96b7b060cb647c6c. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:23.126748 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: This master's current role is: LEARNER
Aug 12 14:52:23 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546343, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.60@KRBTEST.COM
Aug 12 14:52:23 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546343, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
Aug 12 14:52:23 dist-test-slave-6zbq krb5kdc[15164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546343, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.61@KRBTEST.COM
I20260812 14:52:23.133213 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.120541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:39355 (local address 127.4.27.60:44601)
0812 14:52:23.120599 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.120602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.120871 (+ 269us) server_negotiation.cc:407] Connection header received
0812 14:52:23.122509 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.122511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.122523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.122570 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.122929 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123625 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125141 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125434 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125486 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.129782 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.129783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.129783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.129791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.131384 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.132179 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.132181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.132181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.132241 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.132522 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.132523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.132524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.132653 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.132718 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.133038 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.133135 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":23}
I20260812 14:52:23.133375 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.120546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.4.27.3:39355)
0812 14:52:23.120753 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.120794 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.122448 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.122633 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.122633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.122788 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:23.122849 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.122851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123744 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.123745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125063 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.125066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125547 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125689 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.126541 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:23.126546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:23.129678 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:23.131709 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:23.131716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:23.131718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:23.131991 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:23.132312 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:23.132313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:23.132314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:23.132449 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:23.132760 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:23.132767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:23.132913 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.133014 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.133163 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":73,"thread_start_us":110,"threads_started":1}
I20260812 14:52:23.133934 17506 heartbeater.cc:347] Connected to a master server at 127.4.27.60:44601
I20260812 14:52:23.134029 17506 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:23.134204 17506 heartbeater.cc:511] Master 127.4.27.60:44601 requested a full tablet report, sending...
I20260812 14:52:23.134647 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.126589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.4.27.60:44601)
0812 14:52:23.127346 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.127347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.127355 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:23.127374 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.127375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.127384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.127445 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.128117 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.128810 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.129795 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.130048 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.130103 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.131458 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.131459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.131459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.131465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.133221 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.133598 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.133599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.133600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.133658 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.133921 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.133922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.133923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.134060 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.134145 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.134488 (+ 343us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.134563 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733}
I20260812 14:52:23.134680 16671 ts_manager.cc:194] Registered new tserver with Master: dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3:42207)
I20260812 14:52:23.136274 15359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.120801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:53141 (local address 127.4.27.62:46817)
0812 14:52:23.120858 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.120860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.120980 (+ 120us) server_negotiation.cc:407] Connection header received
0812 14:52:23.121811 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.121812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.121824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.121853 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.122855 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123612 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125144 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125426 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125464 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.132394 (+ 6930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.132395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.132395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.132402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.135078 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135444 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.135446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.135446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.135516 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135731 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.135732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.135732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.135847 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.135888 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.136107 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.136174 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:52:23.136344 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.124239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.4.27.61:36581)
0812 14:52:23.126929 (+ 2690us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.126931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.126938 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:23.126958 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.126958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.126970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.127028 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.127730 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.128437 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.129424 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.129716 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.129767 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.132600 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.132601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.132602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.132608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.135086 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135441 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.135442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.135442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.135504 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135671 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.135671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.135672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.135797 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.135886 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.136176 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.136253 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":73,"threads_started":1}
I20260812 14:52:23.136775 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.120546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:53141)
0812 14:52:23.120924 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.120933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.121710 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.121913 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.121913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.122386 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:23.122737 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.122744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123745 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.123746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125042 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.125046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125550 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125689 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.126916 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:23.126921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:23.132332 (+ 5411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:23.135153 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:23.135155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:23.135358 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:23.135606 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:23.135607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:23.135664 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:23.135936 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:23.135937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:23.135976 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.136384 (+ 408us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.136488 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":123,"thread_start_us":103,"threads_started":1}
I20260812 14:52:23.137008 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.120932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:34835 (local address 127.4.27.61:36581)
0812 14:52:23.120977 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.120979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.121022 (+ 43us) server_negotiation.cc:407] Connection header received
0812 14:52:23.121673 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.121675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.121686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.121749 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.122868 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123616 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125152 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125449 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125482 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.131467 (+ 5985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.131468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.131469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.131476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.135699 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.136045 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.136046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.136046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.136100 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.136381 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.136382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.136382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.136491 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.136522 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.136860 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.136933 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:23.137133 17508 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46817
I20260812 14:52:23.137179 17508 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:23.137239 17508 heartbeater.cc:511] Master 127.4.27.62:46817 requested a full tablet report, sending...
I20260812 14:52:23.137324 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.120852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.4.27.3:34835)
0812 14:52:23.120969 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.120978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.121577 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.121861 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.121865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.122451 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:23.122737 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.122744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.123745 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.123746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125068 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.125070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.125553 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.125554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.125688 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.126463 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:23.126473 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:23.131372 (+ 4899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:23.135772 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:23.135773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:23.135773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:23.135963 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:23.136236 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:23.136236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:23.136237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:23.136309 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:23.136641 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:23.136642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:23.136700 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.137111 (+ 411us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.137208 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":377,"thread_start_us":56,"threads_started":1}
I20260812 14:52:23.137473 15247 ts_manager.cc:194] Re-registered known tserver with Master: dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3:42207)
I20260812 14:52:23.137599 17510 heartbeater.cc:347] Connected to a master server at 127.4.27.61:36581
I20260812 14:52:23.137645 17510 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:23.137706 17510 heartbeater.cc:511] Master 127.4.27.61:36581 requested a full tablet report, sending...
I20260812 14:52:23.137876 15247 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:53141
I20260812 14:52:23.137989 16335 ts_manager.cc:194] Registered new tserver with Master: dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3:42207)
I20260812 14:52:23.138726 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.127222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.4.27.62:46817)
0812 14:52:23.127522 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.127524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.127531 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:23.127548 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.127549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.127557 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.127616 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.128858 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.129569 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.130520 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.130815 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.130873 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.133824 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.133825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.133826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.133832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.137438 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.137819 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.137821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.137821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.137876 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.138138 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.138140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.138140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.138292 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.138413 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.138583 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.138656 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":76,"threads_started":1}
I20260812 14:52:23.140146 15247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:23.141721 15247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:23.142386 16600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: NON_VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: true } } }
I20260812 14:52:23.142486 16600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:23.143554 16600 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: attempting to promote NON_VOTER 99b73efbee8b472195586a150bca03cb to VOTER
I20260812 14:52:23.143667 16600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:23.144053 16353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Refusing update from remote peer 129addd4a43b4f1b96b7b060cb647c6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 14:52:23.144084 16688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 LEARNER]: Refusing update from remote peer 129addd4a43b4f1b96b7b060cb647c6c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 14:52:23.144272 16264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 14:52:23.144374 16264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 14:52:23.145073 16265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 99b73efbee8b472195586a150bca03cb (127.4.27.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.145097 16353 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 99b73efbee8b472195586a150bca03cb (127.4.27.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.145795 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.145809 16598 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.145871 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:23.145892 16598 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:52:23.145991 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.146050 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:23.146088 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.146131 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:23.146013 16688 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 99b73efbee8b472195586a150bca03cb (127.4.27.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.147069 16749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "129addd4a43b4f1b96b7b060cb647c6c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:23.147148 16749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:52:23.151960 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.147842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36901 (local address 127.0.0.1:45968)
0812 14:52:23.147995 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.148005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.148054 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.148531 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.148532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.148534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:23.148579 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.148581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.149886 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.149887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.151068 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.151070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.151731 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.151793 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.151797 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.151845 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.151885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260812 14:52:23.152032 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.147772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:33559 (local address 127.0.0.1:36630)
0812 14:52:23.147807 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.147818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.147855 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.148473 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.148474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.148476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:23.148522 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.148524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.149798 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.149799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.150957 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.150960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.151829 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.151881 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.151884 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.151921 (+ 37us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.151980 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":39}
I20260812 14:52:23.152294 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.148152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.4.27.1:33559)
0812 14:52:23.148323 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.148327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.148335 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:23.148369 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.148370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.148384 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.148453 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:23.148604 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.149708 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151049 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.151718 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151824 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.152155 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.152180 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.152217 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260812 14:52:23.152382 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.147948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.4.27.2:36901)
0812 14:52:23.148372 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.148375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.148384 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:23.148415 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.148417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.148430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.148515 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:23.148642 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.149728 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151139 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.151637 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151708 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.151783 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.152186 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.152230 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":98,"threads_started":1}
I20260812 14:52:23.152508 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.148150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.4.27.3:42207)
0812 14:52:23.148554 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.148558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.148714 (+ 156us) server_negotiation.cc:407] Connection header received
0812 14:52:23.148763 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.148765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.148780 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.148846 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:23.149006 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.149999 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.150915 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.151610 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151686 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.151762 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.152288 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.152331 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":108,"threads_started":1}
I20260812 14:52:23.152722 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.147872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42207 (local address 127.0.0.1:56222)
0812 14:52:23.148607 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.148617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.148665 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.148901 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.148902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.148905 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:23.148941 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.148942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.150075 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.150076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.150842 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.150845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.151797 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.151798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.151855 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.151859 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.151904 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.151968 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260812 14:52:23.152961 16875 tablet_service.cc:1467] Processing CreateTablet for tablet e1ef1a8e65434ad3a4334d0445fc6287 (DEFAULT_TABLE table=test-table [id=3cbf9eac5dc64e48880df1b58d27ff12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:23.152961 17133 tablet_service.cc:1467] Processing CreateTablet for tablet e1ef1a8e65434ad3a4334d0445fc6287 (DEFAULT_TABLE table=test-table [id=3cbf9eac5dc64e48880df1b58d27ff12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:23.153334 17133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ef1a8e65434ad3a4334d0445fc6287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:23.153334 16875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ef1a8e65434ad3a4334d0445fc6287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:23.153333 17385 tablet_service.cc:1467] Processing CreateTablet for tablet e1ef1a8e65434ad3a4334d0445fc6287 (DEFAULT_TABLE table=test-table [id=3cbf9eac5dc64e48880df1b58d27ff12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:23.153636 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ef1a8e65434ad3a4334d0445fc6287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:23.156673 17563 tablet_bootstrap.cc:492] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Bootstrap starting.
I20260812 14:52:23.156877 17562 tablet_bootstrap.cc:492] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Bootstrap starting.
I20260812 14:52:23.157281 17563 tablet_bootstrap.cc:654] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:23.157388 17562 tablet_bootstrap.cc:654] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:23.157567 17563 log.cc:824] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:23.157655 17562 log.cc:824] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:23.157883 17564 tablet_bootstrap.cc:492] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Bootstrap starting.
I20260812 14:52:23.158383 17563 tablet_bootstrap.cc:492] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: No bootstrap required, opened a new log
I20260812 14:52:23.158392 17562 tablet_bootstrap.cc:492] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: No bootstrap required, opened a new log
I20260812 14:52:23.158455 17563 ts_tablet_manager.cc:1397] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:23.158581 17562 ts_tablet_manager.cc:1397] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:23.158722 17564 tablet_bootstrap.cc:654] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:23.158990 17564 log.cc:824] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:23.159664 17564 tablet_bootstrap.cc:492] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: No bootstrap required, opened a new log
I20260812 14:52:23.159737 17564 ts_tablet_manager.cc:1397] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:23.160429 17563 raft_consensus.cc:348] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.160586 17563 raft_consensus.cc:374] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:23.160619 17563 raft_consensus.cc:729] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db8e50d4dcf4369bc5473ee39163e7a, State: Initialized, Role: FOLLOWER
I20260812 14:52:23.160749 17563 consensus_queue.cc:260] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.161003 17563 ts_tablet_manager.cc:1428] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:52:23.161110 17013 heartbeater.cc:503] Master 127.4.27.62:46817 was elected leader, sending a full tablet report...
I20260812 14:52:23.161027 17562 raft_consensus.cc:348] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.161172 17562 raft_consensus.cc:374] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:23.161203 17562 raft_consensus.cc:729] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd3cd8bc4464b4088952aef04a0d5df, State: Initialized, Role: FOLLOWER
I20260812 14:52:23.161320 17562 consensus_queue.cc:260] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.161563 17562 ts_tablet_manager.cc:1428] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:52:23.162171 17564 raft_consensus.cc:348] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.162479 17508 heartbeater.cc:503] Master 127.4.27.62:46817 was elected leader, sending a full tablet report...
I20260812 14:52:23.162302 17564 raft_consensus.cc:374] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:23.162967 17564 raft_consensus.cc:729] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a80beb19f348b8a7835bf593fa8c2b, State: Initialized, Role: FOLLOWER
I20260812 14:52:23.163101 17564 consensus_queue.cc:260] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.163342 17564 ts_tablet_manager.cc:1428] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 14:52:23.163394 17257 heartbeater.cc:503] Master 127.4.27.62:46817 was elected leader, sending a full tablet report...
I20260812 14:52:23.191457 17574 raft_consensus.cc:482] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:23.191562 17574 raft_consensus.cc:504] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.191905 17574 leader_election.cc:288] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559), b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:23.195406 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.192038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:33559 (local address 127.4.27.3:34783)
0812 14:52:23.192092 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.192103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.192138 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.192407 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.192408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.192409 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:23.192451 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.192452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.193236 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.193237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.194476 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.194480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.195185 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.195186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.195271 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.195275 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.195316 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.195344 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":26}
I20260812 14:52:23.195425 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.192090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:34783 (local address 127.4.27.1:33559)
0812 14:52:23.192135 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.192137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.192146 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:23.192346 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.192347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.192355 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.192391 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:23.192505 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.193168 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.194558 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.195109 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.195172 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.195182 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.195326 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.195365 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:52:23.195483 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.192057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36901 (local address 127.4.27.3:37559)
0812 14:52:23.192093 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:23.192103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:23.192138 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:23.192316 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:23.192316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:23.192318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:23.192372 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.192373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.193235 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.193235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.194491 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:23.194492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.195251 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.195253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:23.195333 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.195336 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:23.195387 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:52:23.195426 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":40}
I20260812 14:52:23.195510 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.192168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:37559 (local address 127.4.27.2:36901)
0812 14:52:23.192219 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.192221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.192227 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:52:23.192250 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.192251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.192260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.192307 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:23.192438 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.193170 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.194553 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.195108 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.195175 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.195182 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.195410 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.195453 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:23.195756 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef1a8e65434ad3a4334d0445fc6287" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" is_pre_election: true
I20260812 14:52:23.195889 17176 raft_consensus.cc:2454] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 0.
I20260812 14:52:23.196082 17292 leader_election.cc:302] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a80beb19f348b8a7835bf593fa8c2b, dfd3cd8bc4464b4088952aef04a0d5df; no voters:
I20260812 14:52:23.196125 16909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef1a8e65434ad3a4334d0445fc6287" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" is_pre_election: true
I20260812 14:52:23.196194 17574 raft_consensus.cc:2781] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:23.196242 17574 raft_consensus.cc:482] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:23.196272 17574 raft_consensus.cc:3037] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:23.196274 16909 raft_consensus.cc:2454] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 0.
I20260812 14:52:23.196979 17574 raft_consensus.cc:504] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.197096 17574 leader_election.cc:288] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 election: Requested vote from peers 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559), b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:23.197217 16909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef1a8e65434ad3a4334d0445fc6287" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db8e50d4dcf4369bc5473ee39163e7a"
I20260812 14:52:23.197259 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ef1a8e65434ad3a4334d0445fc6287" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a80beb19f348b8a7835bf593fa8c2b"
I20260812 14:52:23.197301 16909 raft_consensus.cc:3037] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:23.197315 17176 raft_consensus.cc:3037] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:23.197907 16909 raft_consensus.cc:2454] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 1.
I20260812 14:52:23.197942 17176 raft_consensus.cc:2454] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 1.
I20260812 14:52:23.198082 17294 leader_election.cc:302] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db8e50d4dcf4369bc5473ee39163e7a, dfd3cd8bc4464b4088952aef04a0d5df; no voters:
I20260812 14:52:23.198171 17574 raft_consensus.cc:2781] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:23.198320 17574 raft_consensus.cc:686] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 LEADER]: Becoming Leader. State: Replica: dfd3cd8bc4464b4088952aef04a0d5df, State: Running, Role: LEADER
I20260812 14:52:23.198426 17574 consensus_queue.cc:237] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } }
I20260812 14:52:23.199214 15246 catalog_manager.cc:5495] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df reported cstate change: term changed from 0 to 1, leader changed from <none> to dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3). New cstate: current_term: 1 leader_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } health_report { overall_health: HEALTHY } } }
I20260812 14:52:23.207670 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.205374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.4.27.3:42207)
0812 14:52:23.205424 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.205426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.205466 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:52:23.205549 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.205550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.205565 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.205633 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:23.205825 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.206257 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.207039 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.207216 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.207264 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.207382 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:23.207464 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:23.207514 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.207609 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.207624 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:52:23.209246 16909 raft_consensus.cc:1261] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Refusing update from remote peer dfd3cd8bc4464b4088952aef04a0d5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:23.209246 17176 raft_consensus.cc:1261] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Refusing update from remote peer dfd3cd8bc4464b4088952aef04a0d5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:23.209565 17577 consensus_queue.cc:1035] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:23.209690 17574 consensus_queue.cc:1035] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:52:23.211006 17260 tablet.cc:2369] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:23.212117 17586 mvcc.cc:205] Tried to move back new op lower bound from 7317693821782720512 to 7317693821741355008. Current Snapshot: MvccSnapshot[applied={T|T < 7317693821782720512}]
I20260812 14:52:23.213737 17591 mvcc.cc:205] Tried to move back new op lower bound from 7317693821782720512 to 7317693821741355008. Current Snapshot: MvccSnapshot[applied={T|T < 7317693821782720512}]
I20260812 14:52:23.215096 15247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:23.215183 15247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:23.216329 15247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 129addd4a43b4f1b96b7b060cb647c6c: Sending DeleteTablet for 3 replicas of tablet e1ef1a8e65434ad3a4334d0445fc6287
I20260812 14:52:23.216626 17385 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ef1a8e65434ad3a4334d0445fc6287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:23 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:56222
I20260812 14:52:23.216634 16875 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ef1a8e65434ad3a4334d0445fc6287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:23 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:36630
I20260812 14:52:23.216821 17133 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ef1a8e65434ad3a4334d0445fc6287 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:23 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:45968
I20260812 14:52:23.217094 17621 tablet_replica.cc:330] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: stopping tablet replica
I20260812 14:52:23.217233 17621 raft_consensus.cc:2229] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:23.217375 17621 raft_consensus.cc:2258] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:23.217406 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 15187
I20260812 14:52:23.251399 17615 tablet_replica.cc:330] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: stopping tablet replica
I20260812 14:52:23.251544 17615 raft_consensus.cc:2229] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:23.251672 17615 raft_consensus.cc:2258] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:23.252220 17621 ts_tablet_manager.cc:1905] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:23.253633 17616 tablet_replica.cc:330] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: stopping tablet replica
I20260812 14:52:23.253679 17621 ts_tablet_manager.cc:1918] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:23.253754 17621 log.cc:1196] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/e1ef1a8e65434ad3a4334d0445fc6287
I20260812 14:52:23.253760 17616 raft_consensus.cc:2229] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:23.253871 17616 raft_consensus.cc:2258] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:23.254148 17621 ts_tablet_manager.cc:1939] T e1ef1a8e65434ad3a4334d0445fc6287 P b7a80beb19f348b8a7835bf593fa8c2b: Deleting consensus metadata
I20260812 14:52:23.254403 17616 ts_tablet_manager.cc:1905] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:23.255962 17616 ts_tablet_manager.cc:1918] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:23.256034 17616 log.cc:1196] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/e1ef1a8e65434ad3a4334d0445fc6287
I20260812 14:52:23.256186 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.217163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.4.27.60:44601)
0812 14:52:23.217200 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.217202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.217451 (+ 249us) server_negotiation.cc:407] Connection header received
0812 14:52:23.249212 (+ 31761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.249213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.249228 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.249276 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.249893 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.250421 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.251314 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.251576 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.251625 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.253158 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.253160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.253160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.253168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.254997 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.255380 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.255381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.255381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.255443 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.255646 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.255647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.255647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.255789 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.255886 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.256011 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.256098 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:52:23.256301 17616 ts_tablet_manager.cc:1939] T e1ef1a8e65434ad3a4334d0445fc6287 P dfd3cd8bc4464b4088952aef04a0d5df: Deleting consensus metadata
W20260812 14:52:23.257539 17292 connection.cc:535] server connection from 127.0.0.1:56222 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 14:52:23.257588 17057 connection.cc:535] server connection from 127.0.0.1:45968 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:52:23.258390 17615 ts_tablet_manager.cc:1905] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:23.259877 17615 ts_tablet_manager.cc:1918] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:23.259955 17615 log.cc:1196] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/e1ef1a8e65434ad3a4334d0445fc6287
I20260812 14:52:23.260219 17615 ts_tablet_manager.cc:1939] T e1ef1a8e65434ad3a4334d0445fc6287 P 3db8e50d4dcf4369bc5473ee39163e7a: Deleting consensus metadata
W20260812 14:52:23.260568 16804 connection.cc:441] server connection from 127.0.0.1:36630 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36630 (request call id 1) could send its response
I20260812 14:52:23.266081 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.259400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.4.27.60:44601)
0812 14:52:23.259435 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.259436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.259481 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:52:23.259533 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.259534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.259543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.259594 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.260449 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.260976 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.261693 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.261886 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.261940 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.263366 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.263367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.263367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.263374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.264938 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.265313 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.265314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.265314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.265373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.265568 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.265569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.265569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.265696 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.265799 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.265916 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.265996 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:52:23.266896 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:23.259360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.4.27.61:36581)
0812 14:52:23.259406 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:23.259408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:23.259416 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:23.259440 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:23.259441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:23.259451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:23.259504 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:23.260308 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.261027 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.262225 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:23.262511 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:23.262562 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:23.263930 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:23.263932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:23.263932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:23.263942 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:23.265532 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.265859 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.265860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.265860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.265915 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:23.266139 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:23.266140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:23.266140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:23.266281 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:23.266408 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:23.266726 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 14:52:23.266806 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:24.213389 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.213124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:47415)
0812 14:52:24.213246 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.213332 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
W20260812 14:52:24.213624 17013 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:46817 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.214721 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.214517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:33377)
0812 14:52:24.214611 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.214679 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
W20260812 14:52:24.214912 17257 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:46817 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.215523 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.215333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:52217)
0812 14:52:24.215434 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.215490 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
W20260812 14:52:24.215679 17508 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:46817 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.716956 17643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 129addd4a43b4f1b96b7b060cb647c6c)
I20260812 14:52:24.717103 17643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:24.717199 17646 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 129addd4a43b4f1b96b7b060cb647c6c)
I20260812 14:52:24.717321 17646 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:24.717386 17643 leader_election.cc:288] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817), d3bcc0c88f914d1eb38695e976d57ecd (127.4.27.61:36581)
I20260812 14:52:24.717712 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.717499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43046)
0812 14:52:24.717623 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.717672 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
I20260812 14:52:24.717733 17646 leader_election.cc:288] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817), 99b73efbee8b472195586a150bca03cb (127.4.27.60:44601)
W20260812 14:52:24.717918 16645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.718024 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.717770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43058)
0812 14:52:24.717929 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.717989 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":43,"threads_started":1}
W20260812 14:52:24.718133 16309 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.718473 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.718334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43068)
0812 14:52:24.718407 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.718443 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260812 14:52:24.718549 16645 leader_election.cc:334] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.718636 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.718552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43076)
0812 14:52:24.718589 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.718610 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:52:24.718742 16309 leader_election.cc:334] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.721009 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.717654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:44601 (local address 127.0.0.1:47990)
0812 14:52:24.717823 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.717833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:24.717885 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:24.718099 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:24.718099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:24.718101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:24.718152 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:24.718154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.719080 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.719081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:24.720231 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:24.720234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.720841 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.720841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:24.720887 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:24.720891 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:24.720930 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 14:52:24.720964 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":50,"threads_started":1}
I20260812 14:52:24.721060 17661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.717817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.4.27.60:44601)
0812 14:52:24.717972 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:24.717974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:24.717980 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:52:24.717999 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:24.718000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:24.718009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:24.718212 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:24.718243 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.719004 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.720312 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.720769 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.720835 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:24.720847 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:24.720945 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:52:24.720978 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":40,"threads_started":1}
I20260812 14:52:24.721349 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.717544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36581 (local address 127.0.0.1:37122)
0812 14:52:24.717762 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.717772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:24.717829 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:24.718012 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:24.718013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:24.718016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:24.718376 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:24.718378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.719365 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.719366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:24.720463 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:24.720466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.721146 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.721147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:24.721227 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:24.721232 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:24.721262 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:52:24.721299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":42,"threads_started":1}
I20260812 14:52:24.721375 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "99b73efbee8b472195586a150bca03cb" is_pre_election: true
I20260812 14:52:24.721483 16688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3bcc0c88f914d1eb38695e976d57ecd in term 1.
I20260812 14:52:24.721567 17652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.717638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.4.27.61:36581)
0812 14:52:24.717885 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:24.717887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:24.717895 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:24.717918 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:24.717919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:24.717928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:24.718441 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:24.718476 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.719293 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.720528 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:24.721034 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:24.721442 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:24.721457 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:24.721475 (+ 18us) server_negotiation.cc:299] Negotiation successful
0812 14:52:24.721514 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":56,"threads_started":1}
I20260812 14:52:24.721614 16302 leader_election.cc:302] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99b73efbee8b472195586a150bca03cb, d3bcc0c88f914d1eb38695e976d57ecd; no voters: 129addd4a43b4f1b96b7b060cb647c6c
I20260812 14:52:24.721714 16353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99b73efbee8b472195586a150bca03cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" is_pre_election: true
I20260812 14:52:24.721753 17646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 14:52:24.721812 16353 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99b73efbee8b472195586a150bca03cb in term 1.
I20260812 14:52:24.721854 17646 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 129addd4a43b4f1b96b7b060cb647c6c)
I20260812 14:52:24.721882 17646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:52:24.721958 16640 leader_election.cc:302] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99b73efbee8b472195586a150bca03cb, d3bcc0c88f914d1eb38695e976d57ecd; no voters: 129addd4a43b4f1b96b7b060cb647c6c
I20260812 14:52:24.722048 17643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 14:52:24.722095 17643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 129addd4a43b4f1b96b7b060cb647c6c)
I20260812 14:52:24.722110 17643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:52:24.722669 17646 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:24.722808 17646 leader_election.cc:288] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 2 election: Requested vote from peers 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817), 99b73efbee8b472195586a150bca03cb (127.4.27.60:44601)
I20260812 14:52:24.722779 17643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:24.722913 17643 leader_election.cc:288] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [CANDIDATE]: Term 2 election: Requested vote from peers 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817), d3bcc0c88f914d1eb38695e976d57ecd (127.4.27.61:36581)
I20260812 14:52:24.722998 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.722898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43086)
0812 14:52:24.722952 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.722976 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 14:52:24.723053 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "99b73efbee8b472195586a150bca03cb"
I20260812 14:52:24.723074 16353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99b73efbee8b472195586a150bca03cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d3bcc0c88f914d1eb38695e976d57ecd"
I20260812 14:52:24.723143 16353 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 99b73efbee8b472195586a150bca03cb in current term 2: Already voted for candidate d3bcc0c88f914d1eb38695e976d57ecd in this term.
I20260812 14:52:24.723111 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.722976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43092)
0812 14:52:24.723009 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.723092 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260812 14:52:24.723142 16688 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d3bcc0c88f914d1eb38695e976d57ecd in current term 2: Already voted for candidate 99b73efbee8b472195586a150bca03cb in this term.
I20260812 14:52:24.723294 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.723189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43106)
0812 14:52:24.723251 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.723274 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 14:52:24.723400 16309 leader_election.cc:334] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.723440 16309 leader_election.cc:302] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3bcc0c88f914d1eb38695e976d57ecd; no voters: 129addd4a43b4f1b96b7b060cb647c6c, 99b73efbee8b472195586a150bca03cb
I20260812 14:52:24.723526 17646 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 14:52:24.723640 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:24.723466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:43110)
0812 14:52:24.723601 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:24.723624 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260812 14:52:24.723703 16645 leader_election.cc:334] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:24.723742 16645 leader_election.cc:302] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99b73efbee8b472195586a150bca03cb; no voters: 129addd4a43b4f1b96b7b060cb647c6c, d3bcc0c88f914d1eb38695e976d57ecd
I20260812 14:52:24.723796 17643 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 14:52:25.214121 17666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:25.213905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:32925)
0812 14:52:25.214035 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:25.214067 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260812 14:52:25.215447 17667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:25.215234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:43323)
0812 14:52:25.215367 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:25.215406 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":51,"threads_started":1}
I20260812 14:52:25.216089 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:25.215916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:47597)
0812 14:52:25.216024 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:25.216055 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260812 14:52:26.214699 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.214495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:39661)
0812 14:52:26.214620 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.214654 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260812 14:52:26.215991 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.215781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:35847)
0812 14:52:26.215933 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.215959 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":52,"threads_started":1}
I20260812 14:52:26.216571 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.216433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:44441)
0812 14:52:26.216519 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.216541 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":44,"threads_started":1}
I20260812 14:52:26.299132 17677 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:26.299221 17677 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:26.299378 17677 leader_election.cc:288] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817), 99b73efbee8b472195586a150bca03cb (127.4.27.60:44601)
I20260812 14:52:26.299576 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "99b73efbee8b472195586a150bca03cb" is_pre_election: true
I20260812 14:52:26.299628 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.299470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:35086)
0812 14:52:26.299560 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.299589 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":44,"threads_started":1}
I20260812 14:52:26.299666 16688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3bcc0c88f914d1eb38695e976d57ecd in term 2.
I20260812 14:52:26.299824 16302 leader_election.cc:302] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99b73efbee8b472195586a150bca03cb, d3bcc0c88f914d1eb38695e976d57ecd; no voters:
I20260812 14:52:26.299924 17677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 14:52:26.299974 17677 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:26.300004 17677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:52:26.300010 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.299893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:35090)
0812 14:52:26.299935 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.299967 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:52:26.300077 16309 leader_election.cc:334] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:26.300738 17677 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:26.300853 17677 leader_election.cc:288] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 3 election: Requested vote from peers 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817), 99b73efbee8b472195586a150bca03cb (127.4.27.60:44601)
I20260812 14:52:26.301020 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.300941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:35106)
0812 14:52:26.300972 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.300997 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260812 14:52:26.301044 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "99b73efbee8b472195586a150bca03cb"
I20260812 14:52:26.301157 16688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:52:26.301360 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.301275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:35118)
0812 14:52:26.301313 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.301339 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:52:26.301432 16309 leader_election.cc:334] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
I20260812 14:52:26.301859 16688 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3bcc0c88f914d1eb38695e976d57ecd in term 3.
I20260812 14:52:26.302024 16302 leader_election.cc:302] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99b73efbee8b472195586a150bca03cb, d3bcc0c88f914d1eb38695e976d57ecd; no voters: 129addd4a43b4f1b96b7b060cb647c6c
I20260812 14:52:26.302127 17677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:52:26.302248 17677 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [term 3 LEADER]: Becoming Leader. State: Replica: d3bcc0c88f914d1eb38695e976d57ecd, State: Running, Role: LEADER
I20260812 14:52:26.302325 17677 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } }
I20260812 14:52:26.302663 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bcc0c88f914d1eb38695e976d57ecd. Latest consensus state: current_term: 3 leader_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:26.302757 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:26.302937 17685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:26.303399 17685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:26.303472 17685 catalog_manager.cc:1259] Loaded cluster ID: 16d04f3c2f684628800b7ec91f2a8a6d
I20260812 14:52:26.303504 17685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:26.303681 17685 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:26.303753 17685 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Loaded TSK: 0
I20260812 14:52:26.303893 17685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:26.330364 16334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:26.331856 16334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:26.332710 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.332616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:35144)
0812 14:52:26.332659 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.332688 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:52:26.332733 16688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [term 3 FOLLOWER]: Refusing update from remote peer d3bcc0c88f914d1eb38695e976d57ecd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 14:52:26.332986 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.332915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.0.0.1:35160)
0812 14:52:26.332943 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.332970 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260812 14:52:26.332926 17677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 14:52:26.333073 16309 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd -> Peer 129addd4a43b4f1b96b7b060cb647c6c (127.4.27.62:46817): Couldn't send request to peer 129addd4a43b4f1b96b7b060cb647c6c. Status: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 14:52:26.333627 17691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3bcc0c88f914d1eb38695e976d57ecd. Latest consensus state: current_term: 3 leader_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:26.333714 17691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:52:26.334089 17677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:26.334180 17677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:26.334259 17691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "129addd4a43b4f1b96b7b060cb647c6c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46817 } } peers { permanent_uuid: "d3bcc0c88f914d1eb38695e976d57ecd" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36581 } attrs { promote: false } } peers { permanent_uuid: "99b73efbee8b472195586a150bca03cb" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 44601 } attrs { promote: false } } }
I20260812 14:52:26.334429 17691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99b73efbee8b472195586a150bca03cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:52:26.338996 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.335956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:33559 (local address 127.0.0.1:52782)
0812 14:52:26.336383 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.336392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:26.336435 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:26.336569 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:26.336570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:26.336571 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:26.336608 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:26.336609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.337402 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.337403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:26.338112 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:26.338114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.338765 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.338766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:26.338838 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.338843 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:26.338886 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:52:26.338927 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":54,"threads_started":1}
I20260812 14:52:26.338996 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.336023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.4.27.1:33559)
0812 14:52:26.336145 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:26.336147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:26.336464 (+ 317us) server_negotiation.cc:407] Connection header received
0812 14:52:26.336492 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:26.336493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:26.336504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:26.336567 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:26.336680 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.337349 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.338188 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.338679 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.338752 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.338765 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:26.338897 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 14:52:26.338932 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20260812 14:52:26.339211 16875 tablet_service.cc:1467] Processing CreateTablet for tablet 38a28aa3cd724fef9b2af8e861abcdf4 (DEFAULT_TABLE table=test-table [id=e12d8a7f543f434282a8144c47e8144d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:26.339344 16875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a28aa3cd724fef9b2af8e861abcdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:26.339339 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.335868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42207 (local address 127.0.0.1:57434)
0812 14:52:26.336020 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.336031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:26.336085 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:26.336266 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:26.336267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:26.336268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:26.336317 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:26.336319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.337214 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.337215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:26.338385 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:26.338388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.339141 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.339142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:26.339203 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.339207 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:26.339233 (+ 26us) client_negotiation.cc:240] Negotiation successful
0812 14:52:26.339274 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":31}
I20260812 14:52:26.339649 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.335923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.4.27.3:42207)
0812 14:52:26.336142 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:26.336144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:26.336152 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:26.336178 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:26.336179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:26.336190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:26.336264 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:26.336387 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.337127 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.338474 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.339028 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.339437 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.339454 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:26.339473 (+ 19us) server_negotiation.cc:299] Negotiation successful
0812 14:52:26.339517 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":60,"threads_started":1}
I20260812 14:52:26.339839 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.335914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36901 (local address 127.0.0.1:47664)
0812 14:52:26.336161 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.336168 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:26.336201 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:26.336338 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:26.336339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:26.336341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:26.336388 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:26.336389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.337197 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.337198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:26.338489 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:26.338491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.339586 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.339587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:26.339654 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.339658 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:26.339725 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:52:26.339765 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":49,"thread_start_us":81,"threads_started":1}
I20260812 14:52:26.340090 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.336100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.4.27.2:36901)
0812 14:52:26.336203 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:26.336205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:26.336213 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:26.336241 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:26.336242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:26.336249 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:26.336323 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:26.336620 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.337116 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.338573 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.339444 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.339502 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.339513 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:26.339884 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 14:52:26.339995 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260812 14:52:26.340360 17133 tablet_service.cc:1467] Processing CreateTablet for tablet 38a28aa3cd724fef9b2af8e861abcdf4 (DEFAULT_TABLE table=test-table [id=e12d8a7f543f434282a8144c47e8144d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:26.340481 17133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a28aa3cd724fef9b2af8e861abcdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:26.340721 17707 tablet_bootstrap.cc:492] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a: Bootstrap starting.
I20260812 14:52:26.341110 17707 tablet_bootstrap.cc:654] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:26.341251 17385 tablet_service.cc:1467] Processing CreateTablet for tablet 38a28aa3cd724fef9b2af8e861abcdf4 (DEFAULT_TABLE table=test-table [id=e12d8a7f543f434282a8144c47e8144d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:26.341351 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a28aa3cd724fef9b2af8e861abcdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:26.341796 17711 tablet_bootstrap.cc:492] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Bootstrap starting.
I20260812 14:52:26.342155 17711 tablet_bootstrap.cc:654] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:26.342430 17712 tablet_bootstrap.cc:492] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Bootstrap starting.
I20260812 14:52:26.342698 17711 tablet_bootstrap.cc:492] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: No bootstrap required, opened a new log
I20260812 14:52:26.342768 17711 ts_tablet_manager.cc:1397] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:26.342928 17711 raft_consensus.cc:348] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.342978 17707 tablet_bootstrap.cc:492] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a: No bootstrap required, opened a new log
I20260812 14:52:26.342985 17712 tablet_bootstrap.cc:654] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:26.343005 17711 raft_consensus.cc:374] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:26.343026 17711 raft_consensus.cc:729] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a80beb19f348b8a7835bf593fa8c2b, State: Initialized, Role: FOLLOWER
I20260812 14:52:26.343019 17707 ts_tablet_manager.cc:1397] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:26.343071 17711 consensus_queue.cc:260] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.343112 17707 raft_consensus.cc:348] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.343173 17707 raft_consensus.cc:374] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:26.343205 17707 raft_consensus.cc:729] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db8e50d4dcf4369bc5473ee39163e7a, State: Initialized, Role: FOLLOWER
I20260812 14:52:26.343271 17711 ts_tablet_manager.cc:1428] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:26.343268 17707 consensus_queue.cc:260] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.343544 17712 tablet_bootstrap.cc:492] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: No bootstrap required, opened a new log
I20260812 14:52:26.343596 17712 ts_tablet_manager.cc:1397] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:26.343708 17707 ts_tablet_manager.cc:1428] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:26.343703 17712 raft_consensus.cc:348] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.343766 17712 raft_consensus.cc:374] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:26.343786 17712 raft_consensus.cc:729] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd3cd8bc4464b4088952aef04a0d5df, State: Initialized, Role: FOLLOWER
I20260812 14:52:26.343827 17712 consensus_queue.cc:260] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.343848 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.343638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:36969)
0812 14:52:26.343790 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.343820 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260812 14:52:26.343981 17712 ts_tablet_manager.cc:1428] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:26.343984 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.343660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:39939)
0812 14:52:26.343928 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.343959 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260812 14:52:26.344135 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.344055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:51559)
0812 14:52:26.344092 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.344113 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:52:26.344391 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.344278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:54691)
0812 14:52:26.344334 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.344365 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 14:52:26.344762 17510 heartbeater.cc:503] Master 127.4.27.61:36581 was elected leader, sending a full tablet report...
I20260812 14:52:26.345054 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.344969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:40609)
0812 14:52:26.345002 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.345031 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 14:52:26.345413 17016 heartbeater.cc:503] Master 127.4.27.61:36581 was elected leader, sending a full tablet report...
I20260812 14:52:26.345479 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.345351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:44711)
0812 14:52:26.345427 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.345452 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 14:52:26.345595 17259 heartbeater.cc:503] Master 127.4.27.61:36581 was elected leader, sending a full tablet report...
W20260812 14:52:26.373082 17511 tablet.cc:2369] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:26.445593 17721 raft_consensus.cc:482] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:26.445693 17721 raft_consensus.cc:504] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.445868 17721 leader_election.cc:288] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559), b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:26.446075 16909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a28aa3cd724fef9b2af8e861abcdf4" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" is_pre_election: true
I20260812 14:52:26.446095 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a28aa3cd724fef9b2af8e861abcdf4" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" is_pre_election: true
I20260812 14:52:26.446173 17176 raft_consensus.cc:2454] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 0.
I20260812 14:52:26.446172 16909 raft_consensus.cc:2454] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 0.
I20260812 14:52:26.446313 17292 leader_election.cc:302] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7a80beb19f348b8a7835bf593fa8c2b, dfd3cd8bc4464b4088952aef04a0d5df; no voters:
I20260812 14:52:26.446442 17721 raft_consensus.cc:2781] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:26.446491 17721 raft_consensus.cc:482] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:26.446504 17721 raft_consensus.cc:3037] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:26.447232 17721 raft_consensus.cc:504] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.447373 17721 leader_election.cc:288] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 election: Requested vote from peers 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559), b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901)
I20260812 14:52:26.447542 16909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a28aa3cd724fef9b2af8e861abcdf4" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3db8e50d4dcf4369bc5473ee39163e7a"
I20260812 14:52:26.447564 17176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a28aa3cd724fef9b2af8e861abcdf4" candidate_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7a80beb19f348b8a7835bf593fa8c2b"
I20260812 14:52:26.447625 17176 raft_consensus.cc:3037] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:26.447620 16909 raft_consensus.cc:3037] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:26.448220 17176 raft_consensus.cc:2454] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 1.
I20260812 14:52:26.448246 16909 raft_consensus.cc:2454] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfd3cd8bc4464b4088952aef04a0d5df in term 1.
I20260812 14:52:26.448390 17294 leader_election.cc:302] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db8e50d4dcf4369bc5473ee39163e7a, dfd3cd8bc4464b4088952aef04a0d5df; no voters:
I20260812 14:52:26.448496 17721 raft_consensus.cc:2781] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:26.448664 17721 raft_consensus.cc:686] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 LEADER]: Becoming Leader. State: Replica: dfd3cd8bc4464b4088952aef04a0d5df, State: Running, Role: LEADER
I20260812 14:52:26.448732 17721 consensus_queue.cc:237] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } }
I20260812 14:52:26.449036 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.448927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:44817)
0812 14:52:26.448968 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.449007 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260812 14:52:26.449429 16335 catalog_manager.cc:5495] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df reported cstate change: term changed from 0 to 1, leader changed from <none> to dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3). New cstate: current_term: 1 leader_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfd3cd8bc4464b4088952aef04a0d5df" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:26.482637 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.480611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.4.27.3:42207)
0812 14:52:26.480657 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:26.480659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:26.480667 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:26.480711 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:26.480712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:26.480722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:26.480765 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:26.480878 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.481352 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.482091 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:26.482288 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:26.482377 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:26.482469 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:26.482495 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:26.482531 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:26.482571 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 14:52:26.482589 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:26.483347 17176 raft_consensus.cc:1261] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Refusing update from remote peer dfd3cd8bc4464b4088952aef04a0d5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:26.483381 16909 raft_consensus.cc:1261] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Refusing update from remote peer dfd3cd8bc4464b4088952aef04a0d5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:26.483642 17726 consensus_queue.cc:1035] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3db8e50d4dcf4369bc5473ee39163e7a" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 33559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:26.483760 17721 consensus_queue.cc:1035] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7a80beb19f348b8a7835bf593fa8c2b" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:26.484287 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.484190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:53053)
0812 14:52:26.484232 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.484259 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:52:26.485069 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.484762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:55235)
0812 14:52:26.485011 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.485042 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231}
I20260812 14:52:26.485195 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.485109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.1:37253)
0812 14:52:26.485140 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.485167 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260812 14:52:26.485361 17733 mvcc.cc:205] Tried to move back new op lower bound from 7317693835194392576 to 7317693835054710784. Current Snapshot: MvccSnapshot[applied={T|T < 7317693835194392576 or (T in {7317693835194392576})}]
I20260812 14:52:26.485881 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.485069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:53671)
0812 14:52:26.485823 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.485854 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":737}
I20260812 14:52:26.486719 17676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.486642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.3:34389)
0812 14:52:26.486670 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.486696 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260812 14:52:26.487291 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.487202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46817 (local address 127.4.27.2:40131)
0812 14:52:26.487241 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.487267 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:46817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260812 14:52:26.487461 16334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:26.487550 16334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:26.488579 16334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3bcc0c88f914d1eb38695e976d57ecd: Sending DeleteTablet for 3 replicas of tablet 38a28aa3cd724fef9b2af8e861abcdf4
I20260812 14:52:26.488849 17133 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a28aa3cd724fef9b2af8e861abcdf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:26 UTC) from {username='oryx', principal='oryx/127.4.27.61@KRBTEST.COM'} at 127.0.0.1:47664
I20260812 14:52:26.488865 16875 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a28aa3cd724fef9b2af8e861abcdf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:26 UTC) from {username='oryx', principal='oryx/127.4.27.61@KRBTEST.COM'} at 127.0.0.1:52782
I20260812 14:52:26.489043 17744 tablet_replica.cc:330] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a: stopping tablet replica
I20260812 14:52:26.489073 17385 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a28aa3cd724fef9b2af8e861abcdf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:26 UTC) from {username='oryx', principal='oryx/127.4.27.61@KRBTEST.COM'} at 127.0.0.1:57434
I20260812 14:52:26.489144 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 16596
I20260812 14:52:26.489105 17744 raft_consensus.cc:2229] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:26.489256 17744 raft_consensus.cc:2258] T 38a28aa3cd724fef9b2af8e861abcdf4 P 3db8e50d4dcf4369bc5473ee39163e7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:26.489354 17745 tablet_replica.cc:330] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: stopping tablet replica
I20260812 14:52:26.489396 17745 raft_consensus.cc:2229] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:26.489428 17745 raft_consensus.cc:2258] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:26.492231 17747 tablet_replica.cc:330] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: stopping tablet replica
I20260812 14:52:26.492282 17747 raft_consensus.cc:2229] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:26.492326 17747 raft_consensus.cc:2258] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:26.496256 17745 ts_tablet_manager.cc:1905] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:26.497361 17747 ts_tablet_manager.cc:1905] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:26.498052 17745 ts_tablet_manager.cc:1918] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:26.498113 17745 log.cc:1196] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/38a28aa3cd724fef9b2af8e861abcdf4
I20260812 14:52:26.498394 17745 ts_tablet_manager.cc:1939] T 38a28aa3cd724fef9b2af8e861abcdf4 P b7a80beb19f348b8a7835bf593fa8c2b: Deleting consensus metadata
I20260812 14:52:26.498627 17747 ts_tablet_manager.cc:1918] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:26.498669 17747 log.cc:1196] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/38a28aa3cd724fef9b2af8e861abcdf4
I20260812 14:52:26.498842 17747 ts_tablet_manager.cc:1939] T 38a28aa3cd724fef9b2af8e861abcdf4 P dfd3cd8bc4464b4088952aef04a0d5df: Deleting consensus metadata
I20260812 14:52:26.499308 16308 catalog_manager.cc:4841] TS dfd3cd8bc4464b4088952aef04a0d5df (127.4.27.3:42207): tablet 38a28aa3cd724fef9b2af8e861abcdf4 (table test-table [id=e12d8a7f543f434282a8144c47e8144d]) successfully deleted
I20260812 14:52:26.499586 16299 catalog_manager.cc:4841] TS b7a80beb19f348b8a7835bf593fa8c2b (127.4.27.2:36901): tablet 38a28aa3cd724fef9b2af8e861abcdf4 (table test-table [id=e12d8a7f543f434282a8144c47e8144d]) successfully deleted
I20260812 14:52:26.507477 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:26.507380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:33559 (local address 127.0.0.1:52798)
0812 14:52:26.507427 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:26.507451 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:33559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:52:26.507583 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 17030
W20260812 14:52:26.507628 16308 catalog_manager.cc:4584] TS 3db8e50d4dcf4369bc5473ee39163e7a (127.4.27.1:33559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38a28aa3cd724fef9b2af8e861abcdf4: Network error: Client connection negotiation failed: client connection to 127.4.27.1:33559: connect: Connection refused (error 111)
I20260812 14:52:26.520553 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 17268
I20260812 14:52:26.532850 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 16083
I20260812 14:52:26.547922 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 16427
2026-08-12T14:52:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6249 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:26 dist-test-slave-6zbq krb5kdc[17753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:26 dist-test-slave-6zbq krb5kdc[17753](info): set up 2 sockets
Aug 12 14:52:26 dist-test-slave-6zbq krb5kdc[17753](info): commencing operation
krb5kdc: starting...
W20260812 14:52:27.748746 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:27 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab.
Entry for principal oryx/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab.
I20260812 14:52:27.854990 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36623
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab
--principal=oryx/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:27.932528 17774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:27.932701 17774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:27.932722 17774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:27.933820 17774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:27.933857 17774 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 14:52:27.933869 17774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:27.933893 17774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:27.933915 17774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:27.933938 17774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:27.935429 17774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36623
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.62.keytab
--principal=oryx/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:27.935642 17774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:27.935837 17774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:27.940244 17790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:27.940313 17774 server_base.cc:1034] running on GCE node
W20260812 14:52:27.940231 17789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:27.940230 17792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:27.940713 17774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:27.940964 17774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:27.942101 17774 hybrid_clock.cc:648] HybridClock initialized: now 1786546347942084 us; error 28 us; skew 500 ppm
Aug 12 14:52:27 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:27.948951 17774 init.cc:377] Logged in from keytab as oryx/127.4.27.62@KRBTEST.COM (short username oryx)
I20260812 14:52:27.949347 17774 webserver.cc:460] Webserver started at http://127.4.27.62:46473/ using document root <none> and password file <none>
I20260812 14:52:27.949549 17774 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:27.949596 17774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:27.949719 17774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:27.950697 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "cc10050a7e5a4477bcbf16b0c5581754"
format_stamp: "Formatted at 2026-08-12 14:52:27 on dist-test-slave-6zbq"
server_key: "9e9aaba20dd2e29a8782d559fbd522b3"
server_key_iv: "9c51bf18ec158ddd77b74666ba2b6a9a"
server_key_version: "encryptionkey@0"
I20260812 14:52:27.950999 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "cc10050a7e5a4477bcbf16b0c5581754"
format_stamp: "Formatted at 2026-08-12 14:52:27 on dist-test-slave-6zbq"
server_key: "9e9aaba20dd2e29a8782d559fbd522b3"
server_key_iv: "9c51bf18ec158ddd77b74666ba2b6a9a"
server_key_version: "encryptionkey@0"
I20260812 14:52:27.952201 17774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:27.952939 17802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:27.953126 17774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:27.953318 17774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "cc10050a7e5a4477bcbf16b0c5581754"
format_stamp: "Formatted at 2026-08-12 14:52:27 on dist-test-slave-6zbq"
server_key: "9e9aaba20dd2e29a8782d559fbd522b3"
server_key_iv: "9c51bf18ec158ddd77b74666ba2b6a9a"
server_key_version: "encryptionkey@0"
I20260812 14:52:27.953385 17774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:27.981550 17774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:27.983206 17774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:27.983335 17774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:27.987751 17774 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:36623
I20260812 14:52:27.987818 17916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:36623 every 8 connection(s)
I20260812 14:52:27.988113 17774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:27.988725 17917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:27.988946 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 17774
I20260812 14:52:27.989045 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:27.989266 4204 external_mini_cluster.cc:1428] Setting key b4b0818827f8c8b0ada8ff73d1ff0899
I20260812 14:52:27.991537 17917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Bootstrap starting.
I20260812 14:52:27.992107 17917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:27.992411 17917 log.cc:824] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:27.993122 17917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: No bootstrap required, opened a new log
Aug 12 14:52:27 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:27.995512 17917 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10050a7e5a4477bcbf16b0c5581754" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36623 } }
I20260812 14:52:27.995652 17917 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:27.995702 17917 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc10050a7e5a4477bcbf16b0c5581754, State: Initialized, Role: FOLLOWER
I20260812 14:52:27.995823 17917 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10050a7e5a4477bcbf16b0c5581754" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36623 } }
I20260812 14:52:27.995882 17917 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:27.995915 17917 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:27.995959 17917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:27.996433 17917 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10050a7e5a4477bcbf16b0c5581754" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36623 } }
I20260812 14:52:27.996527 17917 leader_election.cc:302] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc10050a7e5a4477bcbf16b0c5581754; no voters:
I20260812 14:52:27.996698 17917 leader_election.cc:288] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:27.996732 17930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:27.996867 17930 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [term 1 LEADER]: Becoming Leader. State: Replica: cc10050a7e5a4477bcbf16b0c5581754, State: Running, Role: LEADER
I20260812 14:52:27.996955 17917 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:27.997009 17930 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10050a7e5a4477bcbf16b0c5581754" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36623 } }
I20260812 14:52:27.997359 17934 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc10050a7e5a4477bcbf16b0c5581754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10050a7e5a4477bcbf16b0c5581754" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36623 } } }
I20260812 14:52:27.997359 17930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc10050a7e5a4477bcbf16b0c5581754. Latest consensus state: current_term: 1 leader_uuid: "cc10050a7e5a4477bcbf16b0c5581754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc10050a7e5a4477bcbf16b0c5581754" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36623 } } }
I20260812 14:52:27.997524 17934 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:27.997536 17930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:27.997886 17946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:27.998656 17946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:27.999868 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:27.989750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.4.27.62:36623)
0812 14:52:27.989912 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:27.989921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:27.989944 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:52:27.990363 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:27.990372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:27.990426 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:27.990534 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:27.991294 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:27.992162 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:27.993067 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:27.993494 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:27.993580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:27.996119 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:27.996123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:27.996123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:27.996133 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:27.998058 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:27.998495 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:27.998498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:27.998498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:27.998600 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:27.998825 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:27.998825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:27.998826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:27.999102 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:27.999240 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:27.999469 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 14:52:27.999734 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260812 14:52:28.000418 17946 catalog_manager.cc:1347] Generated new cluster ID: 418d92e2d30849a3a28db5f19938fc82
I20260812 14:52:28.000458 17946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:28.014104 17946 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:28.014930 17946 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:28.021267 17946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Generated new TSK 0
I20260812 14:52:28.021461 17946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab.
Entry for principal oryx/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab.
I20260812 14:52:28.053355 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36623
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:28.132836 17974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:28.133011 17974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:28.133033 17974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:28.134132 17974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:28.134171 17974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:28.134191 17974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:52:28.134222 17974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:28.135735 17974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:28.135938 17974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:28.136139 17974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:28.140815 17986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:28.140918 17974 server_base.cc:1034] running on GCE node
W20260812 14:52:28.140821 17983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:28.140949 17984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:28.141204 17974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:28.141413 17974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:28.142556 17974 hybrid_clock.cc:648] HybridClock initialized: now 1786546348142545 us; error 28 us; skew 500 ppm
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:28.144752 17974 init.cc:377] Logged in from keytab as oryx/127.4.27.1@KRBTEST.COM (short username oryx)
I20260812 14:52:28.145126 17974 webserver.cc:460] Webserver started at http://127.4.27.1:37601/ using document root <none> and password file <none>
I20260812 14:52:28.145335 17974 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:28.145380 17974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:28.145498 17974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:28.146492 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "1658c8617d0f46d8b1f78584bd14e13b"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "331a00bda1ba1c7551c1e906c7956a63"
server_key_iv: "7636a334e260d4c212ec1c43fbf5ae69"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.146804 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "1658c8617d0f46d8b1f78584bd14e13b"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "331a00bda1ba1c7551c1e906c7956a63"
server_key_iv: "7636a334e260d4c212ec1c43fbf5ae69"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.147953 17974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:28.148578 17998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.148718 17974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:28.148797 17974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "1658c8617d0f46d8b1f78584bd14e13b"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "331a00bda1ba1c7551c1e906c7956a63"
server_key_iv: "7636a334e260d4c212ec1c43fbf5ae69"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.148854 17974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:28.155893 17974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:28.157433 17974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:28.157549 17974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:28.157970 17974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:28.158011 17974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.158036 17974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:28.158053 17974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.164674 17974 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:32891
I20260812 14:52:28.164737 18220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:32891 every 8 connection(s)
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:28.165024 17974 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:28.167706 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.158365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:38129 (local address 127.4.27.62:36623)
0812 14:52:28.158413 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.158415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.158665 (+ 250us) server_negotiation.cc:407] Connection header received
0812 14:52:28.159224 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.159225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.159235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.159274 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.160044 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.160804 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.162280 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.162568 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.162623 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.165394 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.165396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.165396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.165403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.166477 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.166836 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.166837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.166838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.166896 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.167108 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.167109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.167109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.167226 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.167280 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.167525 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.167615 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:52:28.167830 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.158373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.4.27.1:38129)
0812 14:52:28.158530 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.158578 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.159133 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.159343 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.159347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.159671 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.159953 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.159960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.160911 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.160912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.162191 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.162194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.162644 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.162645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.162795 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.163697 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.163706 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.165314 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.166545 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.166552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.166554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.166769 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.166950 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.166950 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.166950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.167052 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.167282 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.167289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.167410 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.167496 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.167632 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260812 14:52:28.168268 18221 heartbeater.cc:347] Connected to a master server at 127.4.27.62:36623
I20260812 14:52:28.168345 18221 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:28.168506 18221 heartbeater.cc:511] Master 127.4.27.62:36623 requested a full tablet report, sending...
I20260812 14:52:28.169018 17828 ts_manager.cc:194] Registered new tserver with Master: 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891)
I20260812 14:52:28.169162 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 17974
I20260812 14:52:28.169272 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:28.169464 4204 external_mini_cluster.cc:1428] Setting key 19302a978b90365f7bebc32cedbf4049
I20260812 14:52:28.169837 17828 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:38129
I20260812 14:52:28.174405 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.169219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.4.27.1:54009)
0812 14:52:28.169381 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.169391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.169451 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.169573 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.169574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.169773 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.169827 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.169829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.170644 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.170645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.171140 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.171142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.171463 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.171464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.171523 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.171894 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.171898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.172140 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.173547 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.173548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.173548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.173697 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.173879 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.173879 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.173880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.173923 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.174129 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.174130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.174161 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.174200 (+ 39us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.174292 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260812 14:52:28.174437 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.169165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:54009 (local address 127.4.27.62:36623)
0812 14:52:28.169203 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.169205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.169455 (+ 250us) server_negotiation.cc:407] Connection header received
0812 14:52:28.169500 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.169501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.169511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.169564 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.169909 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.170548 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.171209 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.171406 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.171457 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.172217 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.172219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.172219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.172226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.173473 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.173767 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.173768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.173768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.173825 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.173971 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.173972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.173972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.174078 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.174131 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.174249 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.174333 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab.
Entry for principal oryx/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab.
I20260812 14:52:28.192097 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36623
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:28.272768 18236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:28.272943 18236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:28.272965 18236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:28.274016 18236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:28.274047 18236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:28.274065 18236 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:52:28.274087 18236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:28.275533 18236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:28.275707 18236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:28.275872 18236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:52:28.280460 18236 server_base.cc:1034] running on GCE node
W20260812 14:52:28.280444 18242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:28.280416 18243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:28.280407 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:28.280745 18236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:28.280957 18236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:28.282088 18236 hybrid_clock.cc:648] HybridClock initialized: now 1786546348282077 us; error 29 us; skew 500 ppm
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:28.284561 18236 init.cc:377] Logged in from keytab as oryx/127.4.27.2@KRBTEST.COM (short username oryx)
I20260812 14:52:28.284960 18236 webserver.cc:460] Webserver started at http://127.4.27.2:46211/ using document root <none> and password file <none>
I20260812 14:52:28.285173 18236 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:28.285231 18236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:28.285336 18236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:28.286386 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "19b5b4341f48483cab4ffe423d82a1d3"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "5819c18540ff840442266414b42eed00"
server_key_iv: "c50bf615381b8461710c89a12ac8df10"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.286710 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "19b5b4341f48483cab4ffe423d82a1d3"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "5819c18540ff840442266414b42eed00"
server_key_iv: "c50bf615381b8461710c89a12ac8df10"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.287889 18236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:28.288519 18260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.288668 18236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:28.288753 18236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "19b5b4341f48483cab4ffe423d82a1d3"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "5819c18540ff840442266414b42eed00"
server_key_iv: "c50bf615381b8461710c89a12ac8df10"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.288813 18236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:28.299393 18236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:28.301304 18236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:28.301429 18236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:28.301859 18236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:28.301896 18236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.301923 18236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:28.301939 18236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:28.308640 18236 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:33607
I20260812 14:52:28.308701 18456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:33607 every 8 connection(s)
I20260812 14:52:28.308986 18236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:28.310964 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.302245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:53643 (local address 127.4.27.62:36623)
0812 14:52:28.302308 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.302310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.302531 (+ 221us) server_negotiation.cc:407] Connection header received
0812 14:52:28.303123 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.303124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.303133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.303173 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.303970 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.304743 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.306185 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.306497 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.306559 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.308555 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.308557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.308557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.308566 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.309923 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.310249 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.310249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.310250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.310293 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.310498 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.310498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.310498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.310593 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.310640 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.310841 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.310903 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:52:28.311079 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.302217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.4.27.2:53643)
0812 14:52:28.302399 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.302448 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.303032 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.303246 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.303251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.303573 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.303862 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.303868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.304830 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.304831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.306104 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.306107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.306589 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.306590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.306725 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.307118 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.307125 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.308451 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.309987 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.309992 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.309994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.310193 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.310330 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.310330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.310331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.310452 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.310639 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.310642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.310748 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.310821 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.310929 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260812 14:52:28.311520 18457 heartbeater.cc:347] Connected to a master server at 127.4.27.62:36623
I20260812 14:52:28.311599 18457 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:28.311776 18457 heartbeater.cc:511] Master 127.4.27.62:36623 requested a full tablet report, sending...
I20260812 14:52:28.312410 17829 ts_manager.cc:194] Registered new tserver with Master: 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607)
I20260812 14:52:28.312795 17829 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:53643
I20260812 14:52:28.316164 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 18236
I20260812 14:52:28.316263 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:28.316471 4204 external_mini_cluster.cc:1428] Setting key 7233ebaf6ad5ae2e680c4e3e9e04c72a
I20260812 14:52:28.317417 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.312504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:40585 (local address 127.4.27.62:36623)
0812 14:52:28.312540 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.312542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.312972 (+ 430us) server_negotiation.cc:407] Connection header received
0812 14:52:28.313014 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.313015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.313025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.313070 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.313393 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.313836 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.314474 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.314639 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.314682 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.315326 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.315327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.315327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.315333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.316330 (+ 997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.316651 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.316652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.316653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.316724 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.316893 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.316894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.316894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.317069 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.317128 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.317268 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.317336 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:28.317431 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.312450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.4.27.2:40585)
0812 14:52:28.312922 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.312931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.312978 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.313082 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.313083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.313277 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.313332 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.313334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.313907 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.313908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.314420 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.314422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.314687 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.314688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.314727 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.315013 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.315016 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.315262 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.316391 (+ 1129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.316391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.316392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.316583 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.316776 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.316777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.316778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.316841 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.317121 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.317122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.317164 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.317237 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.317333 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab.
Entry for principal oryx/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab.
I20260812 14:52:28.337982 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36623
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:28.416592 18474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:28.416778 18474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:28.416800 18474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:28.417901 18474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:28.417939 18474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:28.417959 18474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 14:52:28.417990 18474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:28.419450 18474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.27.62:36623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:28.419660 18474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:28.419853 18474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:28.424338 18481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:28.424330 18480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:28.424331 18484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:28.424595 18474 server_base.cc:1034] running on GCE node
I20260812 14:52:28.424779 18474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:28.424983 18474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:28.426126 18474 hybrid_clock.cc:648] HybridClock initialized: now 1786546348426105 us; error 33 us; skew 500 ppm
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:28.428949 18474 init.cc:377] Logged in from keytab as oryx/127.4.27.3@KRBTEST.COM (short username oryx)
I20260812 14:52:28.429328 18474 webserver.cc:460] Webserver started at http://127.4.27.3:35749/ using document root <none> and password file <none>
I20260812 14:52:28.429527 18474 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:28.429574 18474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:28.429703 18474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:28.430707 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "bf240d905b654134a1c1a790149ff421"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "1f9283ac1a3fb5d097049b99c953448c"
server_key_iv: "83187b809977e39100f42c7e9f2c12fe"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.431033 18474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "bf240d905b654134a1c1a790149ff421"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "1f9283ac1a3fb5d097049b99c953448c"
server_key_iv: "83187b809977e39100f42c7e9f2c12fe"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.432201 18474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:28.432806 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.432950 18474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:28.433029 18474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "bf240d905b654134a1c1a790149ff421"
format_stamp: "Formatted at 2026-08-12 14:52:28 on dist-test-slave-6zbq"
server_key: "1f9283ac1a3fb5d097049b99c953448c"
server_key_iv: "83187b809977e39100f42c7e9f2c12fe"
server_key_version: "encryptionkey@0"
I20260812 14:52:28.433092 18474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:28.459416 18474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:28.461325 18474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:28.461450 18474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:28.461905 18474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:28.461951 18474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.461978 18474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:28.461995 18474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.468066 18474 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:34295
I20260812 14:52:28.468127 18705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:34295 every 8 connection(s)
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:28.468425 18474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:28.470980 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.462310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:49045 (local address 127.4.27.62:36623)
0812 14:52:28.462377 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.462380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.462539 (+ 159us) server_negotiation.cc:407] Connection header received
0812 14:52:28.462972 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.462973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.462986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.463050 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.463783 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.464529 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.465951 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.466241 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.466310 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.468758 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.468759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.468760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.468767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.469938 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.470280 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.470280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.470281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.470325 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.470537 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.470538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.470538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.470631 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.470680 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.470863 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.470920 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:52:28.471094 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.462260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.4.27.3:49045)
0812 14:52:28.462423 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.462460 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.462886 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.463115 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.463119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.463423 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.463687 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.463693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.464627 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.464628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.465864 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.465867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.466328 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.466330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.466496 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.466962 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.466972 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.468651 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.470004 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.470009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.470011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.470223 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.470401 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.470402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.470402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.470493 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.470674 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.470676 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.470775 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.470842 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.470948 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260812 14:52:28.471558 18707 heartbeater.cc:347] Connected to a master server at 127.4.27.62:36623
I20260812 14:52:28.471634 18707 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:28.471793 18707 heartbeater.cc:511] Master 127.4.27.62:36623 requested a full tablet report, sending...
I20260812 14:52:28.472164 17828 ts_manager.cc:194] Registered new tserver with Master: bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.472236 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 18474
I20260812 14:52:28.472339 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:28.472548 4204 external_mini_cluster.cc:1428] Setting key 35b8a98630159ffabd2eb1b3e3796ea6
I20260812 14:52:28.472625 17828 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:49045
I20260812 14:52:28.473363 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:52:28.477681 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.472420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.4.27.3:34401)
0812 14:52:28.472717 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.472730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.472805 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.472952 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.472953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.473100 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.473144 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.473145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.473825 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.473826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.474322 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.474324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.474670 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.474671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.474730 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.475073 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.475078 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.475398 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.476856 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.476857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.476857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.476996 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.477178 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.477179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.477179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.477246 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.477474 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.477474 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.477507 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.477550 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.477617 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":74,"threads_started":1}
I20260812 14:52:28.477741 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.472471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:34401 (local address 127.4.27.62:36623)
0812 14:52:28.472506 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.472507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.472780 (+ 273us) server_negotiation.cc:407] Connection header received
0812 14:52:28.472855 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.472856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.472866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.472914 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.473214 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.473742 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.474417 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.474614 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.474666 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.475483 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.475485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.475485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.475490 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.476779 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.477058 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.477059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.477059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.477118 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.477304 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.477305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.477305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.477417 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.477479 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.477573 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.477648 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:52:28.494633 17927 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:28.489697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.4.27.62:36623)
0812 14:52:28.489745 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.489746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.489758 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:52:28.489831 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.489832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.489841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.489884 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.490441 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.490889 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.491664 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.491837 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.491890 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.494569 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41078: BlockingRecv error: recv got EOF from 127.0.0.1:41078 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:28.500751 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.495243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.4.27.62:36623)
0812 14:52:28.495299 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.495301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.495348 (+ 47us) server_negotiation.cc:407] Connection header received
0812 14:52:28.495396 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.495396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.495403 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.495436 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.495799 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.496224 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.496790 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.496947 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.496986 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.498816 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.498817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.498817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.498823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.500025 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.500285 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.500286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.500286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.500330 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.500436 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.500436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.500436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.500514 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.500577 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.500641 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.500694 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 14:52:28.501927 17828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:28.503463 17828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:28.509343 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.505337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34295 (local address 127.0.0.1:55654)
0812 14:52:28.505792 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.505802 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.505859 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.506254 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.506255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.506257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:28.506308 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.506309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.507491 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.507492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.508391 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.508393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.509035 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.509036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.509131 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.509135 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.509195 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.509263 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":31,"thread_start_us":54,"threads_started":1}
I20260812 14:52:28.509351 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.505427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.4.27.3:34295)
0812 14:52:28.505763 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.505766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.505864 (+ 98us) server_negotiation.cc:407] Connection header received
0812 14:52:28.506098 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.506101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.506116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.506192 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:28.506434 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.507424 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.508492 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.508957 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.509020 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.509092 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.509216 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.509254 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260812 14:52:28.509351 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.505319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:32891 (local address 127.0.0.1:55316)
0812 14:52:28.505443 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.505480 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.505540 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.505937 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.505940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.505942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:28.506003 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.506005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.507256 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.507257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.508374 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.508377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.509032 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.509033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.509110 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.509124 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.509198 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.509269 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":31,"thread_start_us":45,"threads_started":1}
I20260812 14:52:28.509637 18749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.505633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.4.27.1:32891)
0812 14:52:28.505784 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.505788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.505798 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:28.505837 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.505839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.505854 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.505927 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:28.506066 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.507177 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.508460 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.508950 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.509012 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.509088 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.509497 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.509541 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260812 14:52:28.510002 18589 tablet_service.cc:1467] Processing CreateTablet for tablet fa6405d529584d3192156fb085cfc16e (DEFAULT_TABLE table=test-table [id=d7b32140ed43454384a4c7ee2879246d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:28.510403 18746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.505506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:33607 (local address 127.0.0.1:42094)
0812 14:52:28.505806 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.505815 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.505867 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.506063 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.506064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.506066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:28.506114 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.506116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.507089 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.507090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.508375 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.508377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.510167 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.510168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.510231 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.510236 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.510285 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.510323 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":11,"thread_start_us":38,"threads_started":1}
I20260812 14:52:28.510444 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6405d529584d3192156fb085cfc16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.510710 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.505610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.4.27.2:33607)
0812 14:52:28.505741 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.505746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.505922 (+ 176us) server_negotiation.cc:407] Connection header received
0812 14:52:28.505963 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.505965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.505979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.506240 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:28.506300 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.507015 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.509127 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.510077 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.510145 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.510218 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.510307 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.510606 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260812 14:52:28.510738 18090 tablet_service.cc:1467] Processing CreateTablet for tablet fa6405d529584d3192156fb085cfc16e (DEFAULT_TABLE table=test-table [id=d7b32140ed43454384a4c7ee2879246d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:28.511060 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6405d529584d3192156fb085cfc16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.511340 18336 tablet_service.cc:1467] Processing CreateTablet for tablet fa6405d529584d3192156fb085cfc16e (DEFAULT_TABLE table=test-table [id=d7b32140ed43454384a4c7ee2879246d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:28.511686 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa6405d529584d3192156fb085cfc16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.513872 18750 tablet_bootstrap.cc:492] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Bootstrap starting.
I20260812 14:52:28.514518 18750 tablet_bootstrap.cc:654] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.514546 18754 tablet_bootstrap.cc:492] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Bootstrap starting.
I20260812 14:52:28.514848 18750 log.cc:824] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:28.515213 18754 tablet_bootstrap.cc:654] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.515399 18759 tablet_bootstrap.cc:492] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Bootstrap starting.
I20260812 14:52:28.515538 18750 tablet_bootstrap.cc:492] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: No bootstrap required, opened a new log
I20260812 14:52:28.515532 18754 log.cc:824] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:28.515609 18750 ts_tablet_manager.cc:1397] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:28.516084 18759 tablet_bootstrap.cc:654] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.516117 18754 tablet_bootstrap.cc:492] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: No bootstrap required, opened a new log
I20260812 14:52:28.516172 18754 ts_tablet_manager.cc:1397] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:28.516388 18759 log.cc:824] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:28.517122 18759 tablet_bootstrap.cc:492] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: No bootstrap required, opened a new log
I20260812 14:52:28.517192 18759 ts_tablet_manager.cc:1397] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:28.517477 18750 raft_consensus.cc:348] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.517639 18750 raft_consensus.cc:374] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.517670 18750 raft_consensus.cc:729] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1658c8617d0f46d8b1f78584bd14e13b, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.517637 18754 raft_consensus.cc:348] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.517860 18754 raft_consensus.cc:374] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.517869 18750 consensus_queue.cc:260] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.517918 18754 raft_consensus.cc:729] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf240d905b654134a1c1a790149ff421, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.518046 18754 consensus_queue.cc:260] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.518128 18750 ts_tablet_manager.cc:1428] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:52:28.518270 18221 heartbeater.cc:503] Master 127.4.27.62:36623 was elected leader, sending a full tablet report...
I20260812 14:52:28.518640 18754 ts_tablet_manager.cc:1428] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:28.518669 18707 heartbeater.cc:503] Master 127.4.27.62:36623 was elected leader, sending a full tablet report...
I20260812 14:52:28.519012 18759 raft_consensus.cc:348] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.519143 18759 raft_consensus.cc:374] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.519177 18759 raft_consensus.cc:729] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.519279 18759 consensus_queue.cc:260] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.519517 18759 ts_tablet_manager.cc:1428] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:52:28.519656 18457 heartbeater.cc:503] Master 127.4.27.62:36623 was elected leader, sending a full tablet report...
W20260812 14:52:28.559530 18458 tablet.cc:2369] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:28.603730 18769 raft_consensus.cc:482] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:28.603821 18769 raft_consensus.cc:504] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.604146 18769 leader_election.cc:288] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.607714 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.604312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:32891 (local address 127.4.27.2:42847)
0812 14:52:28.604424 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.604434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.604484 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.604636 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.604637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.604639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:28.604711 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.604713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.605606 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.605606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.606802 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.606805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.607470 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.607471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.607549 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.607553 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.607608 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.607644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 14:52:28.607743 18749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.604376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:42847 (local address 127.4.27.1:32891)
0812 14:52:28.604421 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.604423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.604471 (+ 48us) server_negotiation.cc:407] Connection header received
0812 14:52:28.604535 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.604536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.604547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.604591 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:28.604793 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.605529 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.606906 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.607393 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.607468 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.607480 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.607626 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.607669 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:28.608291 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6405d529584d3192156fb085cfc16e" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b" is_pre_election: true
I20260812 14:52:28.608449 18126 raft_consensus.cc:2454] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:28.608705 18262 leader_election.cc:302] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, 19b5b4341f48483cab4ffe423d82a1d3; no voters:
I20260812 14:52:28.608814 18769 raft_consensus.cc:2781] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:28.608878 18769 raft_consensus.cc:482] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:28.608898 18769 raft_consensus.cc:3037] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.609827 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.604424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:55801 (local address 127.4.27.3:34295)
0812 14:52:28.604490 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.604492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.606773 (+ 2281us) server_negotiation.cc:407] Connection header received
0812 14:52:28.606867 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.606868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.606880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.606909 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:28.607109 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.607654 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.608942 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.609447 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.609514 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.609526 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.609742 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.609781 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:28.609843 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.604527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34295 (local address 127.4.27.2:55801)
0812 14:52:28.606706 (+ 2179us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.606717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.606802 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.606968 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.606969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.606970 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:28.607039 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.607040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.607763 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.607764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.608841 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.608844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.609580 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.609582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.609655 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.609659 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.609717 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.609754 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2158,"thread_start_us":120,"threads_started":1}
I20260812 14:52:28.610029 18769 raft_consensus.cc:504] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.610162 18769 leader_election.cc:288] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Requested vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.610220 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6405d529584d3192156fb085cfc16e" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421" is_pre_election: true
I20260812 14:52:28.610316 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6405d529584d3192156fb085cfc16e" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b"
I20260812 14:52:28.610384 18629 raft_consensus.cc:2454] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:28.610412 18126 raft_consensus.cc:3037] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.610599 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa6405d529584d3192156fb085cfc16e" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421"
I20260812 14:52:28.610670 18629 raft_consensus.cc:3037] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.611122 18126 raft_consensus.cc:2454] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:28.611295 18262 leader_election.cc:302] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, 19b5b4341f48483cab4ffe423d82a1d3; no voters:
I20260812 14:52:28.611310 18629 raft_consensus.cc:2454] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:28.611456 18769 raft_consensus.cc:2781] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:28.611625 18769 raft_consensus.cc:686] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Becoming Leader. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Running, Role: LEADER
I20260812 14:52:28.611717 18769 consensus_queue.cc:237] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.612442 17828 catalog_manager.cc:5495] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:28.623445 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.621287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.4.27.2:33607)
0812 14:52:28.621326 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.621328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.621336 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:28.621399 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.621400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.621416 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.621473 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:28.621584 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.622000 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.622776 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.622970 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.623020 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.623163 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:28.623261 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:28.623324 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.623369 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.623387 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:52:28.624851 18629 raft_consensus.cc:1261] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:28.624863 18126 raft_consensus.cc:1261] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:28.625137 18769 consensus_queue.cc:1035] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:28.625264 18772 consensus_queue.cc:1035] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:28.627272 18778 mvcc.cc:205] Tried to move back new op lower bound from 7317693843965353984 to 7317693843914092544. Current Snapshot: MvccSnapshot[applied={T|T < 7317693843965353984}]
I20260812 14:52:28.627367 18780 mvcc.cc:205] Tried to move back new op lower bound from 7317693843965353984 to 7317693843914092544. Current Snapshot: MvccSnapshot[applied={T|T < 7317693843965353984}]
I20260812 14:52:28.628041 18779 mvcc.cc:205] Tried to move back new op lower bound from 7317693843965353984 to 7317693843914092544. Current Snapshot: MvccSnapshot[applied={T|T < 7317693843965353984 or (T in {7317693843965353984})}]
I20260812 14:52:28.629522 17829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:28.629612 17829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:28.630257 17829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Sending DeleteTablet for 3 replicas of tablet fa6405d529584d3192156fb085cfc16e
I20260812 14:52:28.630563 18090 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6405d529584d3192156fb085cfc16e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:28 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260812 14:52:28.630580 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6405d529584d3192156fb085cfc16e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:28 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55654
I20260812 14:52:28.630609 18336 tablet_service.cc:1514] Processing DeleteTablet for tablet fa6405d529584d3192156fb085cfc16e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:28 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42094
I20260812 14:52:28.630779 18798 tablet_replica.cc:330] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: stopping tablet replica
I20260812 14:52:28.630865 18798 raft_consensus.cc:2229] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:28.631098 18801 tablet_replica.cc:330] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: stopping tablet replica
I20260812 14:52:28.631078 17829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 14:52:28.631201 18798 raft_consensus.cc:2258] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 14:52:28.631213 17829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:28.631553 18801 raft_consensus.cc:2229] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:28.631742 18798 ts_tablet_manager.cc:1905] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:28.632503 18090 tablet_service.cc:1467] Processing CreateTablet for tablet 0d39c6845da6476aafdccbbe217b85c3 (DEFAULT_TABLE table=test-table [id=fdff66b29a8f488ca370bab67904620f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:28.632532 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 0d39c6845da6476aafdccbbe217b85c3 (DEFAULT_TABLE table=test-table [id=fdff66b29a8f488ca370bab67904620f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:28.632607 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d39c6845da6476aafdccbbe217b85c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.632637 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d39c6845da6476aafdccbbe217b85c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.632543 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 0d39c6845da6476aafdccbbe217b85c3 (DEFAULT_TABLE table=test-table [id=fdff66b29a8f488ca370bab67904620f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:28.632931 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d39c6845da6476aafdccbbe217b85c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.633703 18750 tablet_bootstrap.cc:492] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Bootstrap starting.
I20260812 14:52:28.634006 18754 tablet_bootstrap.cc:492] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Bootstrap starting.
I20260812 14:52:28.634059 18750 tablet_bootstrap.cc:654] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.634183 18759 tablet_bootstrap.cc:492] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Bootstrap starting.
I20260812 14:52:28.634433 18754 tablet_bootstrap.cc:654] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.634565 18759 tablet_bootstrap.cc:654] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.635264 18798 ts_tablet_manager.cc:1918] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:28.635298 18759 tablet_bootstrap.cc:492] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: No bootstrap required, opened a new log
I20260812 14:52:28.635334 18798 log.cc:1196] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/fa6405d529584d3192156fb085cfc16e
I20260812 14:52:28.635334 18759 ts_tablet_manager.cc:1397] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:28.635476 18759 raft_consensus.cc:348] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.635533 18759 raft_consensus.cc:374] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.635552 18759 raft_consensus.cc:729] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.635591 18798 ts_tablet_manager.cc:1939] T fa6405d529584d3192156fb085cfc16e P 1658c8617d0f46d8b1f78584bd14e13b: Deleting consensus metadata
I20260812 14:52:28.635592 18759 consensus_queue.cc:260] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.635672 18759 ts_tablet_manager.cc:1428] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.635942 17804 catalog_manager.cc:4841] TS 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891): tablet fa6405d529584d3192156fb085cfc16e (table test-table [id=d7b32140ed43454384a4c7ee2879246d]) successfully deleted
I20260812 14:52:28.636134 18750 tablet_bootstrap.cc:492] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: No bootstrap required, opened a new log
I20260812 14:52:28.636178 18750 ts_tablet_manager.cc:1397] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:28.636322 18750 raft_consensus.cc:348] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.636391 18750 raft_consensus.cc:374] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.636415 18750 raft_consensus.cc:729] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1658c8617d0f46d8b1f78584bd14e13b, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.636451 18750 consensus_queue.cc:260] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.636533 18750 ts_tablet_manager.cc:1428] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.636806 18801 raft_consensus.cc:2258] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:28.637210 18801 ts_tablet_manager.cc:1905] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:28.638407 18801 ts_tablet_manager.cc:1918] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:28.638479 18801 log.cc:1196] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/fa6405d529584d3192156fb085cfc16e
I20260812 14:52:28.638545 18799 tablet_replica.cc:330] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: stopping tablet replica
I20260812 14:52:28.638664 18799 raft_consensus.cc:2229] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:28.638732 18754 tablet_bootstrap.cc:492] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: No bootstrap required, opened a new log
I20260812 14:52:28.638747 18801 ts_tablet_manager.cc:1939] T fa6405d529584d3192156fb085cfc16e P 19b5b4341f48483cab4ffe423d82a1d3: Deleting consensus metadata
I20260812 14:52:28.638789 18754 ts_tablet_manager.cc:1397] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 14:52:28.638895 18799 raft_consensus.cc:2258] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:28.638928 18754 raft_consensus.cc:348] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.638995 18754 raft_consensus.cc:374] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.639020 18754 raft_consensus.cc:729] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf240d905b654134a1c1a790149ff421, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.639057 18754 consensus_queue.cc:260] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.639083 17804 catalog_manager.cc:4841] TS 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607): tablet fa6405d529584d3192156fb085cfc16e (table test-table [id=d7b32140ed43454384a4c7ee2879246d]) successfully deleted
I20260812 14:52:28.639138 18754 ts_tablet_manager.cc:1428] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:28.639452 18799 ts_tablet_manager.cc:1905] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:28.640559 18799 ts_tablet_manager.cc:1918] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:28.640625 18799 log.cc:1196] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/fa6405d529584d3192156fb085cfc16e
I20260812 14:52:28.640827 18799 ts_tablet_manager.cc:1939] T fa6405d529584d3192156fb085cfc16e P bf240d905b654134a1c1a790149ff421: Deleting consensus metadata
I20260812 14:52:28.641167 17803 catalog_manager.cc:4841] TS bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295): tablet fa6405d529584d3192156fb085cfc16e (table test-table [id=d7b32140ed43454384a4c7ee2879246d]) successfully deleted
W20260812 14:52:28.665711 18222 tablet.cc:2369] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:28.708472 18788 raft_consensus.cc:482] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:28.708563 18788 raft_consensus.cc:504] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.708751 18788 leader_election.cc:288] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.708961 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d39c6845da6476aafdccbbe217b85c3" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421" is_pre_election: true
I20260812 14:52:28.708957 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d39c6845da6476aafdccbbe217b85c3" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b" is_pre_election: true
I20260812 14:52:28.709048 18629 raft_consensus.cc:2454] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:28.709069 18126 raft_consensus.cc:2454] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:28.709218 18261 leader_election.cc:302] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5b4341f48483cab4ffe423d82a1d3, bf240d905b654134a1c1a790149ff421; no voters:
I20260812 14:52:28.709321 18788 raft_consensus.cc:2781] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:28.709381 18788 raft_consensus.cc:482] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:28.709409 18788 raft_consensus.cc:3037] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.710374 18788 raft_consensus.cc:504] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.710523 18788 leader_election.cc:288] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Requested vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.710671 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d39c6845da6476aafdccbbe217b85c3" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421"
I20260812 14:52:28.710768 18629 raft_consensus.cc:3037] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.711608 18629 raft_consensus.cc:2454] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:28.711745 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d39c6845da6476aafdccbbe217b85c3" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b"
I20260812 14:52:28.711829 18126 raft_consensus.cc:3037] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.711822 18261 leader_election.cc:302] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5b4341f48483cab4ffe423d82a1d3, bf240d905b654134a1c1a790149ff421; no voters:
I20260812 14:52:28.711948 18788 raft_consensus.cc:2781] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:28.711994 18788 raft_consensus.cc:686] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Becoming Leader. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Running, Role: LEADER
I20260812 14:52:28.712050 18788 consensus_queue.cc:237] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.712637 17828 catalog_manager.cc:5495] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:28.712774 18126 raft_consensus.cc:2454] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
W20260812 14:52:28.718914 18708 tablet.cc:2369] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.62@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:28.757249 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.750944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36623 (local address 127.0.0.1:41092)
0812 14:52:28.751102 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.751112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.751167 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.751298 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.751298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.751578 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:28.751624 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.751626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.752255 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.752256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.752813 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.752815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.753095 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.753095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.753131 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.753614 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:28.753619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:28.754965 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:28.756287 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.756295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.756298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.756479 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.756663 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:28.756664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:28.756664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:28.756717 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:28.756915 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:28.756923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:28.756968 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.757026 (+ 58us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.757139 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260812 14:52:28.757301 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.751001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.4.27.62:36623)
0812 14:52:28.751050 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.751052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.751167 (+ 115us) server_negotiation.cc:407] Connection header received
0812 14:52:28.751209 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.751210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.751218 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.751287 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.751683 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.752183 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.752873 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.753047 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.753092 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.755071 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:28.755072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:28.755073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:28.755079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:28.756226 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.756558 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.756559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.756559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.756611 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:28.756766 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:28.756766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:28.756767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:28.756863 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:28.756917 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.757056 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.757203 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":26}
I20260812 14:52:28.759123 17828 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:41092:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 14:52:28.759334 17828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:28.760775 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 344a6f7e32e44c11906be27fd3964fc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d221c3ab6a0459194e85c9e3bb9c69c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:28.760798 18090 tablet_service.cc:1467] Processing CreateTablet for tablet 344a6f7e32e44c11906be27fd3964fc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d221c3ab6a0459194e85c9e3bb9c69c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:28.760787 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 344a6f7e32e44c11906be27fd3964fc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d221c3ab6a0459194e85c9e3bb9c69c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 14:52:28.760927 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344a6f7e32e44c11906be27fd3964fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.760931 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344a6f7e32e44c11906be27fd3964fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.760931 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344a6f7e32e44c11906be27fd3964fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:28.761976 18759 tablet_bootstrap.cc:492] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: Bootstrap starting.
I20260812 14:52:28.762269 18759 tablet_bootstrap.cc:654] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.762382 18750 tablet_bootstrap.cc:492] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: Bootstrap starting.
I20260812 14:52:28.762382 18754 tablet_bootstrap.cc:492] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: Bootstrap starting.
I20260812 14:52:28.762766 18754 tablet_bootstrap.cc:654] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.762766 18750 tablet_bootstrap.cc:654] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:28.763415 18750 tablet_bootstrap.cc:492] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: No bootstrap required, opened a new log
I20260812 14:52:28.763471 18750 ts_tablet_manager.cc:1397] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:28.763545 18759 tablet_bootstrap.cc:492] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: No bootstrap required, opened a new log
I20260812 14:52:28.763553 18754 tablet_bootstrap.cc:492] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: No bootstrap required, opened a new log
I20260812 14:52:28.763597 18759 ts_tablet_manager.cc:1397] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:52:28.763595 18754 ts_tablet_manager.cc:1397] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:28.763612 18750 raft_consensus.cc:348] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.763684 18750 raft_consensus.cc:374] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.763705 18750 raft_consensus.cc:729] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1658c8617d0f46d8b1f78584bd14e13b, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.763711 18759 raft_consensus.cc:348] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.763751 18750 consensus_queue.cc:260] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.763777 18759 raft_consensus.cc:374] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.763737 18754 raft_consensus.cc:348] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.763797 18759 raft_consensus.cc:729] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.763799 18754 raft_consensus.cc:374] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:28.763819 18754 raft_consensus.cc:729] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf240d905b654134a1c1a790149ff421, State: Initialized, Role: FOLLOWER
I20260812 14:52:28.763834 18759 consensus_queue.cc:260] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.763851 18750 ts_tablet_manager.cc:1428] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:28.763862 18754 consensus_queue.cc:260] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.763913 18759 ts_tablet_manager.cc:1428] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:28.763938 18754 ts_tablet_manager.cc:1428] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:28.763983 18763 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.764068 18763 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.764072 18764 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.764160 18764 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.764147 18772 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.764221 18772 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.920521 18772 raft_consensus.cc:482] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:28.920619 18772 raft_consensus.cc:504] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.920784 18772 leader_election.cc:288] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.920997 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a6f7e32e44c11906be27fd3964fc5" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b" is_pre_election: true
I20260812 14:52:28.920993 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a6f7e32e44c11906be27fd3964fc5" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421" is_pre_election: true
I20260812 14:52:28.921119 18629 raft_consensus.cc:2454] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:28.921116 18126 raft_consensus.cc:2454] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:28.921272 18261 leader_election.cc:302] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5b4341f48483cab4ffe423d82a1d3, bf240d905b654134a1c1a790149ff421; no voters:
I20260812 14:52:28.921391 18772 raft_consensus.cc:2781] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:28.921443 18772 raft_consensus.cc:482] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:28.921458 18772 raft_consensus.cc:3037] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.922147 18772 raft_consensus.cc:504] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.922258 18772 leader_election.cc:288] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Requested vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:28.922422 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a6f7e32e44c11906be27fd3964fc5" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421"
I20260812 14:52:28.922420 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344a6f7e32e44c11906be27fd3964fc5" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b"
I20260812 14:52:28.922500 18126 raft_consensus.cc:3037] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.922500 18629 raft_consensus.cc:3037] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:28.923189 18629 raft_consensus.cc:2454] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:28.923197 18126 raft_consensus.cc:2454] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:28.923360 18262 leader_election.cc:302] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, 19b5b4341f48483cab4ffe423d82a1d3; no voters:
I20260812 14:52:28.923457 18772 raft_consensus.cc:2781] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:28.923519 18772 raft_consensus.cc:686] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Becoming Leader. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Running, Role: LEADER
I20260812 14:52:28.923574 18772 consensus_queue.cc:237] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:28.923842 18788 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: TxnStatusTablet state changed. Reason: New leader 19b5b4341f48483cab4ffe423d82a1d3. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.923928 18788 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:28.924085 18825 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 14:52:28.924221 18825 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 14:52:28.924132 17827 catalog_manager.cc:5495] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:28.946436 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.944006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.4.27.2:33607)
0812 14:52:28.944044 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.944046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.944094 (+ 48us) server_negotiation.cc:407] Connection header received
0812 14:52:28.944179 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.944180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.944188 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.944232 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:28.944363 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.944817 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.945843 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.946011 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.946075 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.946233 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:28.946257 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:28.946286 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.946371 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.946390 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:52:28.946497 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.943970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:33607 (local address 127.0.0.1:42106)
0812 14:52:28.944053 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.944063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.944113 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.944255 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.944256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.944257 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:28.944308 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.944310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.944880 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.944881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.945775 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.945778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.946073 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.946074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.946138 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.946156 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:28.946304 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:28.946309 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.946371 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.946418 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 14:52:28.947336 18629 raft_consensus.cc:1261] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:28.947352 18126 raft_consensus.cc:1261] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:28.947528 18788 consensus_queue.cc:1035] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:28.947618 18772 consensus_queue.cc:1035] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:28.948038 18764 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: TxnStatusTablet state changed. Reason: New leader 19b5b4341f48483cab4ffe423d82a1d3. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.948077 18763 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: TxnStatusTablet state changed. Reason: New leader 19b5b4341f48483cab4ffe423d82a1d3. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.948140 18764 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.948158 18763 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.948716 18788 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.948788 18788 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:28.948864 18788 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.948912 18788 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 19b5b4341f48483cab4ffe423d82a1d3: This TxnStatusTablet replica's current role is: LEADER
I20260812 14:52:28.949329 18763 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.949414 18763 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P 1658c8617d0f46d8b1f78584bd14e13b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.949626 18764 tablet_replica.cc:439] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } } }
I20260812 14:52:28.949704 18764 tablet_replica.cc:442] T 344a6f7e32e44c11906be27fd3964fc5 P bf240d905b654134a1c1a790149ff421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 14:52:28.954922 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.952842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:37899 (local address 127.4.27.2:33607)
0812 14:52:28.952873 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.952875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.952994 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:52:28.953048 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.953048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.953056 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.953103 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:28.953209 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.953629 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.954293 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.954520 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.954571 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.954709 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:28.954739 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:28.954778 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:28.954832 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 14:52:28.954851 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":14}
I20260812 14:52:28.954977 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:28.952801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:33607 (local address 127.4.27.2:37899)
0812 14:52:28.952939 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:28.952948 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:28.952998 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:28.953108 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:28.953108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:28.953110 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 14:52:28.953145 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.953146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.953690 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.953691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.954229 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:28.954231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.954578 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.954578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:28.954627 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.954639 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 14:52:28.954782 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 14:52:28.954784 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:28.954829 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:52:28.954875 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260812 14:52:28.956930 18126 raft_consensus.cc:1261] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:28.957008 18629 raft_consensus.cc:1261] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:28.957139 18772 consensus_queue.cc:1035] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:28.957324 18788 consensus_queue.cc:1035] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:28.970281 17829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:28.970402 17829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:28.970989 17829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Sending DeleteTablet for 3 replicas of tablet 0d39c6845da6476aafdccbbe217b85c3
I20260812 14:52:28.971251 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d39c6845da6476aafdccbbe217b85c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:28 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55654
I20260812 14:52:28.971355 18799 tablet_replica.cc:330] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: stopping tablet replica
I20260812 14:52:28.971365 18335 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d39c6845da6476aafdccbbe217b85c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:28 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42094
I20260812 14:52:28.971413 18799 raft_consensus.cc:2229] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:28.971390 18090 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d39c6845da6476aafdccbbe217b85c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:28 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260812 14:52:28.971436 18801 tablet_replica.cc:330] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: stopping tablet replica
I20260812 14:52:28.971454 18799 raft_consensus.cc:2258] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:28.971475 18801 raft_consensus.cc:2229] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:28.971519 18801 raft_consensus.cc:2258] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:28.971527 18798 tablet_replica.cc:330] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: stopping tablet replica
I20260812 14:52:28.971585 18798 raft_consensus.cc:2229] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:28.971616 18798 raft_consensus.cc:2258] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:28.971864 18801 ts_tablet_manager.cc:1905] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:28.971889 18798 ts_tablet_manager.cc:1905] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:28.973450 18799 ts_tablet_manager.cc:1905] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:28.973500 18798 ts_tablet_manager.cc:1918] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:28.973544 18798 log.cc:1196] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/0d39c6845da6476aafdccbbe217b85c3
I20260812 14:52:28.973532 18801 ts_tablet_manager.cc:1918] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:28.973587 18801 log.cc:1196] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/0d39c6845da6476aafdccbbe217b85c3
I20260812 14:52:28.973743 18798 ts_tablet_manager.cc:1939] T 0d39c6845da6476aafdccbbe217b85c3 P 1658c8617d0f46d8b1f78584bd14e13b: Deleting consensus metadata
I20260812 14:52:28.973759 18801 ts_tablet_manager.cc:1939] T 0d39c6845da6476aafdccbbe217b85c3 P 19b5b4341f48483cab4ffe423d82a1d3: Deleting consensus metadata
I20260812 14:52:28.974025 17804 catalog_manager.cc:4841] TS 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607): tablet 0d39c6845da6476aafdccbbe217b85c3 (table test-table [id=fdff66b29a8f488ca370bab67904620f]) successfully deleted
I20260812 14:52:28.974102 17804 catalog_manager.cc:4841] TS 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891): tablet 0d39c6845da6476aafdccbbe217b85c3 (table test-table [id=fdff66b29a8f488ca370bab67904620f]) successfully deleted
I20260812 14:52:28.974925 18799 ts_tablet_manager.cc:1918] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:28.974987 18799 log.cc:1196] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/0d39c6845da6476aafdccbbe217b85c3
I20260812 14:52:28.975152 18799 ts_tablet_manager.cc:1939] T 0d39c6845da6476aafdccbbe217b85c3 P bf240d905b654134a1c1a790149ff421: Deleting consensus metadata
I20260812 14:52:28.975437 17803 catalog_manager.cc:4841] TS bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295): tablet 0d39c6845da6476aafdccbbe217b85c3 (table test-table [id=fdff66b29a8f488ca370bab67904620f]) successfully deleted
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:28 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:52:28.999481 17927 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:28.993906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.4.27.62:36623)
0812 14:52:28.993952 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:28.993955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:28.994083 (+ 128us) server_negotiation.cc:407] Connection header received
0812 14:52:28.994152 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:28.994152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:28.994162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:28.994206 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:28.994888 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.995362 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.996371 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:28.996540 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:28.996581 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:28.999426 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41108: BlockingRecv error: recv got EOF from 127.0.0.1:41108 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 12 14:52:29 dist-test-slave-6zbq krb5kdc[17753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM
I20260812 14:52:29.006330 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.000172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.4.27.62:36623)
0812 14:52:29.000210 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:29.000212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:29.000264 (+ 52us) server_negotiation.cc:407] Connection header received
0812 14:52:29.000323 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:29.000324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:29.000333 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:29.000376 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:29.000796 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.001257 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.001881 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.002067 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.002125 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.004070 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:29.004072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:29.004072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:29.004079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:29.005374 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:29.005677 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:29.005678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:29.005678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:29.005731 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:29.005896 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:29.005897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:29.005897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:29.005998 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:29.006079 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:29.006182 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 14:52:29.006251 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:29.007684 17829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:29.007833 17829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:29.009119 18589 tablet_service.cc:1467] Processing CreateTablet for tablet ec467ad8064c428ab8f5eb5bc0678ffc (DEFAULT_TABLE table=test-table [id=c1927f7c4afd4175844bdcf71a2ab702]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:29.009181 18335 tablet_service.cc:1467] Processing CreateTablet for tablet ec467ad8064c428ab8f5eb5bc0678ffc (DEFAULT_TABLE table=test-table [id=c1927f7c4afd4175844bdcf71a2ab702]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:29.009191 18090 tablet_service.cc:1467] Processing CreateTablet for tablet ec467ad8064c428ab8f5eb5bc0678ffc (DEFAULT_TABLE table=test-table [id=c1927f7c4afd4175844bdcf71a2ab702]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:29.009258 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec467ad8064c428ab8f5eb5bc0678ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:29.009295 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec467ad8064c428ab8f5eb5bc0678ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:29.009295 18335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec467ad8064c428ab8f5eb5bc0678ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:29.010565 18759 tablet_bootstrap.cc:492] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Bootstrap starting.
I20260812 14:52:29.010643 18750 tablet_bootstrap.cc:492] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Bootstrap starting.
I20260812 14:52:29.010857 18759 tablet_bootstrap.cc:654] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:29.011063 18750 tablet_bootstrap.cc:654] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:29.011404 18754 tablet_bootstrap.cc:492] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Bootstrap starting.
I20260812 14:52:29.011408 18759 tablet_bootstrap.cc:492] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: No bootstrap required, opened a new log
I20260812 14:52:29.011461 18759 ts_tablet_manager.cc:1397] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:52:29.011564 18759 raft_consensus.cc:348] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.011648 18759 raft_consensus.cc:374] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:29.011675 18759 raft_consensus.cc:729] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Initialized, Role: FOLLOWER
I20260812 14:52:29.011713 18759 consensus_queue.cc:260] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.011785 18754 tablet_bootstrap.cc:654] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:29.011801 18759 ts_tablet_manager.cc:1428] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:29.012408 18750 tablet_bootstrap.cc:492] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: No bootstrap required, opened a new log
I20260812 14:52:29.012465 18750 ts_tablet_manager.cc:1397] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:29.012622 18750 raft_consensus.cc:348] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.012699 18750 raft_consensus.cc:374] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:29.012727 18750 raft_consensus.cc:729] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1658c8617d0f46d8b1f78584bd14e13b, State: Initialized, Role: FOLLOWER
I20260812 14:52:29.012773 18750 consensus_queue.cc:260] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.012876 18750 ts_tablet_manager.cc:1428] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:29.013432 18754 tablet_bootstrap.cc:492] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: No bootstrap required, opened a new log
I20260812 14:52:29.013477 18754 ts_tablet_manager.cc:1397] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:52:29.013620 18754 raft_consensus.cc:348] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.013676 18754 raft_consensus.cc:374] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:29.013695 18754 raft_consensus.cc:729] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf240d905b654134a1c1a790149ff421, State: Initialized, Role: FOLLOWER
I20260812 14:52:29.013733 18754 consensus_queue.cc:260] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.013818 18754 ts_tablet_manager.cc:1428] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:29.050232 18763 raft_consensus.cc:482] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:29.050329 18763 raft_consensus.cc:504] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.050664 18763 leader_election.cc:288] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:29.054096 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.050989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:33229 (local address 127.4.27.3:34295)
0812 14:52:29.051032 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:29.051034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:29.051065 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:52:29.051137 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:29.051138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:29.051147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:29.051199 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:29.051353 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.051834 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.053282 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.053768 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.053838 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.053850 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:29.053997 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 14:52:29.054033 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:29.054116 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.050813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34295 (local address 127.4.27.1:33229)
0812 14:52:29.051007 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:29.051018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:29.051080 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:29.051236 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:29.051237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:29.051238 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:29.051298 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:29.051299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.051916 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.051917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:29.053190 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:29.053193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.053865 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.053866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:29.053933 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.053938 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:52:29.053988 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:52:29.054027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
I20260812 14:52:29.054435 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.050907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:60057 (local address 127.4.27.2:33607)
0812 14:52:29.050952 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:29.050954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:29.051008 (+ 54us) server_negotiation.cc:407] Connection header received
0812 14:52:29.051118 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:29.051119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:29.051131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:29.051180 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:29.051322 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.052073 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.053529 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.053925 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.053995 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.054005 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:29.054307 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 14:52:29.054372 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:29.054426 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.050811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:33607 (local address 127.4.27.1:60057)
0812 14:52:29.050950 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:29.050960 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:29.051037 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:29.051183 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:29.051184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:29.051186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:29.051262 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:29.051264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.052164 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.052164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:29.053458 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:29.053460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.054049 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.054050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:29.054127 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.054131 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:29.054279 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 14:52:29.054320 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260812 14:52:29.054668 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec467ad8064c428ab8f5eb5bc0678ffc" candidate_uuid: "1658c8617d0f46d8b1f78584bd14e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421" is_pre_election: true
I20260812 14:52:29.054761 18629 raft_consensus.cc:2454] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1658c8617d0f46d8b1f78584bd14e13b in term 0.
I20260812 14:52:29.054931 18002 leader_election.cc:302] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, bf240d905b654134a1c1a790149ff421; no voters:
I20260812 14:52:29.054900 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec467ad8064c428ab8f5eb5bc0678ffc" candidate_uuid: "1658c8617d0f46d8b1f78584bd14e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b5b4341f48483cab4ffe423d82a1d3" is_pre_election: true
I20260812 14:52:29.055016 18370 raft_consensus.cc:2454] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1658c8617d0f46d8b1f78584bd14e13b in term 0.
I20260812 14:52:29.055044 18763 raft_consensus.cc:2781] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:29.055094 18763 raft_consensus.cc:482] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:29.055114 18763 raft_consensus.cc:3037] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:29.055868 18763 raft_consensus.cc:504] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.055987 18763 leader_election.cc:288] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [CANDIDATE]: Term 1 election: Requested vote from peers 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:29.056124 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec467ad8064c428ab8f5eb5bc0678ffc" candidate_uuid: "1658c8617d0f46d8b1f78584bd14e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421"
I20260812 14:52:29.056186 18629 raft_consensus.cc:3037] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:29.056238 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec467ad8064c428ab8f5eb5bc0678ffc" candidate_uuid: "1658c8617d0f46d8b1f78584bd14e13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b5b4341f48483cab4ffe423d82a1d3"
I20260812 14:52:29.056303 18370 raft_consensus.cc:3037] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:29.056808 18629 raft_consensus.cc:2454] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1658c8617d0f46d8b1f78584bd14e13b in term 1.
I20260812 14:52:29.056941 18370 raft_consensus.cc:2454] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1658c8617d0f46d8b1f78584bd14e13b in term 1.
I20260812 14:52:29.056959 18002 leader_election.cc:302] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, bf240d905b654134a1c1a790149ff421; no voters:
I20260812 14:52:29.057053 18763 raft_consensus.cc:2781] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:29.057245 18763 raft_consensus.cc:686] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 1 LEADER]: Becoming Leader. State: Replica: 1658c8617d0f46d8b1f78584bd14e13b, State: Running, Role: LEADER
I20260812 14:52:29.057320 18763 consensus_queue.cc:237] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.058107 17827 catalog_manager.cc:5495] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1). New cstate: current_term: 1 leader_uuid: "1658c8617d0f46d8b1f78584bd14e13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:29.072779 18749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.070525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.4.27.1:32891)
0812 14:52:29.070568 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:29.070570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:29.070580 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:29.070653 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:29.070654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:29.070673 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:29.070732 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:29.070830 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.071339 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.072081 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.072287 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.072340 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.072484 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:29.072584 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:29.072648 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:29.072694 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 14:52:29.072714 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:52:29.073585 18629 raft_consensus.cc:1261] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Refusing update from remote peer 1658c8617d0f46d8b1f78584bd14e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:29.073685 18370 raft_consensus.cc:1261] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Refusing update from remote peer 1658c8617d0f46d8b1f78584bd14e13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:29.073832 18875 consensus_queue.cc:1035] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:29.073937 18763 consensus_queue.cc:1035] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:29.077168 17829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:29.077265 17829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:29.077805 17829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Sending DeleteTablet for 3 replicas of tablet ec467ad8064c428ab8f5eb5bc0678ffc
I20260812 14:52:29.078011 18090 tablet_service.cc:1514] Processing DeleteTablet for tablet ec467ad8064c428ab8f5eb5bc0678ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:29 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260812 14:52:29.078110 18335 tablet_service.cc:1514] Processing DeleteTablet for tablet ec467ad8064c428ab8f5eb5bc0678ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:29 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42094
I20260812 14:52:29.078191 18798 tablet_replica.cc:330] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: stopping tablet replica
I20260812 14:52:29.078197 18801 tablet_replica.cc:330] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: stopping tablet replica
I20260812 14:52:29.078236 18798 raft_consensus.cc:2229] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:29.078246 18801 raft_consensus.cc:2229] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:29.078285 18801 raft_consensus.cc:2258] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:29.078310 18798 raft_consensus.cc:2258] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:29.078401 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet ec467ad8064c428ab8f5eb5bc0678ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:29 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55654
I20260812 14:52:29.078523 18799 tablet_replica.cc:330] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: stopping tablet replica
I20260812 14:52:29.078568 18799 raft_consensus.cc:2229] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:29.078603 18799 raft_consensus.cc:2258] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:29.078603 18801 ts_tablet_manager.cc:1905] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:29.078994 18798 ts_tablet_manager.cc:1905] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:29.078964 17829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:29.079093 17829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:29.080013 18801 ts_tablet_manager.cc:1918] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:29.080062 18801 log.cc:1196] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/ec467ad8064c428ab8f5eb5bc0678ffc
I20260812 14:52:29.080271 18801 ts_tablet_manager.cc:1939] T ec467ad8064c428ab8f5eb5bc0678ffc P 19b5b4341f48483cab4ffe423d82a1d3: Deleting consensus metadata
I20260812 14:52:29.080329 18798 ts_tablet_manager.cc:1918] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:29.080372 18798 log.cc:1196] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/ec467ad8064c428ab8f5eb5bc0678ffc
I20260812 14:52:29.080544 18798 ts_tablet_manager.cc:1939] T ec467ad8064c428ab8f5eb5bc0678ffc P 1658c8617d0f46d8b1f78584bd14e13b: Deleting consensus metadata
I20260812 14:52:29.080844 17804 catalog_manager.cc:4841] TS 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607): tablet ec467ad8064c428ab8f5eb5bc0678ffc (table test-table [id=c1927f7c4afd4175844bdcf71a2ab702]) successfully deleted
I20260812 14:52:29.080909 17804 catalog_manager.cc:4841] TS 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891): tablet ec467ad8064c428ab8f5eb5bc0678ffc (table test-table [id=c1927f7c4afd4175844bdcf71a2ab702]) successfully deleted
I20260812 14:52:29.081557 18090 tablet_service.cc:1467] Processing CreateTablet for tablet 2784542f635346999bbe9acfc20751fb (DEFAULT_TABLE table=test-table [id=b1cc29d89e87488ab87a1fdb471629cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:29.081563 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 2784542f635346999bbe9acfc20751fb (DEFAULT_TABLE table=test-table [id=b1cc29d89e87488ab87a1fdb471629cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:29.081575 18335 tablet_service.cc:1467] Processing CreateTablet for tablet 2784542f635346999bbe9acfc20751fb (DEFAULT_TABLE table=test-table [id=b1cc29d89e87488ab87a1fdb471629cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:29.081662 18090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2784542f635346999bbe9acfc20751fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:29.081662 18335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2784542f635346999bbe9acfc20751fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:29.081676 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2784542f635346999bbe9acfc20751fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:29.082778 18750 tablet_bootstrap.cc:492] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Bootstrap starting.
I20260812 14:52:29.082932 18754 tablet_bootstrap.cc:492] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Bootstrap starting.
I20260812 14:52:29.083110 18750 tablet_bootstrap.cc:654] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:29.083344 18759 tablet_bootstrap.cc:492] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Bootstrap starting.
I20260812 14:52:29.083556 18754 tablet_bootstrap.cc:654] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:29.083631 18759 tablet_bootstrap.cc:654] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:29.084041 18750 tablet_bootstrap.cc:492] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: No bootstrap required, opened a new log
I20260812 14:52:29.084072 18750 ts_tablet_manager.cc:1397] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:29.084173 18750 raft_consensus.cc:348] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.084213 18750 raft_consensus.cc:374] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:29.084224 18750 raft_consensus.cc:729] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1658c8617d0f46d8b1f78584bd14e13b, State: Initialized, Role: FOLLOWER
I20260812 14:52:29.084260 18750 consensus_queue.cc:260] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.084339 18750 ts_tablet_manager.cc:1428] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:29.084507 18759 tablet_bootstrap.cc:492] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: No bootstrap required, opened a new log
I20260812 14:52:29.084544 18759 ts_tablet_manager.cc:1397] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:52:29.084616 18759 raft_consensus.cc:348] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.084654 18759 raft_consensus.cc:374] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:29.084667 18759 raft_consensus.cc:729] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Initialized, Role: FOLLOWER
I20260812 14:52:29.084710 18759 consensus_queue.cc:260] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.084800 18759 ts_tablet_manager.cc:1428] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:29.085311 18799 ts_tablet_manager.cc:1905] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:29.086397 18799 ts_tablet_manager.cc:1918] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:29.086458 18799 log.cc:1196] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/ec467ad8064c428ab8f5eb5bc0678ffc
I20260812 14:52:29.086656 18799 ts_tablet_manager.cc:1939] T ec467ad8064c428ab8f5eb5bc0678ffc P bf240d905b654134a1c1a790149ff421: Deleting consensus metadata
I20260812 14:52:29.086920 17803 catalog_manager.cc:4841] TS bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295): tablet ec467ad8064c428ab8f5eb5bc0678ffc (table test-table [id=c1927f7c4afd4175844bdcf71a2ab702]) successfully deleted
I20260812 14:52:29.086980 18754 tablet_bootstrap.cc:492] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: No bootstrap required, opened a new log
I20260812 14:52:29.087026 18754 ts_tablet_manager.cc:1397] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260812 14:52:29.087150 18754 raft_consensus.cc:348] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.087222 18754 raft_consensus.cc:374] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:29.087247 18754 raft_consensus.cc:729] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf240d905b654134a1c1a790149ff421, State: Initialized, Role: FOLLOWER
I20260812 14:52:29.087292 18754 consensus_queue.cc:260] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.087373 18754 ts_tablet_manager.cc:1428] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:29.203002 18772 raft_consensus.cc:482] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:29.203100 18772 raft_consensus.cc:504] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.203269 18772 leader_election.cc:288] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:29.203455 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2784542f635346999bbe9acfc20751fb" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b" is_pre_election: true
I20260812 14:52:29.203464 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2784542f635346999bbe9acfc20751fb" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421" is_pre_election: true
I20260812 14:52:29.203540 18126 raft_consensus.cc:2454] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:29.203559 18629 raft_consensus.cc:2454] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 0.
I20260812 14:52:29.203701 18262 leader_election.cc:302] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, 19b5b4341f48483cab4ffe423d82a1d3; no voters:
I20260812 14:52:29.203809 18772 raft_consensus.cc:2781] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:29.203859 18772 raft_consensus.cc:482] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:29.203873 18772 raft_consensus.cc:3037] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:29.204532 18772 raft_consensus.cc:504] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.204638 18772 leader_election.cc:288] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Requested vote from peers 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891), bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295)
I20260812 14:52:29.204813 18126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2784542f635346999bbe9acfc20751fb" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1658c8617d0f46d8b1f78584bd14e13b"
I20260812 14:52:29.204841 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2784542f635346999bbe9acfc20751fb" candidate_uuid: "19b5b4341f48483cab4ffe423d82a1d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf240d905b654134a1c1a790149ff421"
I20260812 14:52:29.204890 18126 raft_consensus.cc:3037] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:29.204909 18629 raft_consensus.cc:3037] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:29.205437 18126 raft_consensus.cc:2454] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:29.205520 18629 raft_consensus.cc:2454] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5b4341f48483cab4ffe423d82a1d3 in term 1.
I20260812 14:52:29.205631 18262 leader_election.cc:302] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1658c8617d0f46d8b1f78584bd14e13b, 19b5b4341f48483cab4ffe423d82a1d3; no voters:
I20260812 14:52:29.205715 18772 raft_consensus.cc:2781] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:29.205767 18772 raft_consensus.cc:686] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Becoming Leader. State: Replica: 19b5b4341f48483cab4ffe423d82a1d3, State: Running, Role: LEADER
I20260812 14:52:29.205816 18772 consensus_queue.cc:237] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } }
I20260812 14:52:29.206315 17829 catalog_manager.cc:5495] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "19b5b4341f48483cab4ffe423d82a1d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19b5b4341f48483cab4ffe423d82a1d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:29.234608 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.232815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.4.27.2:33607)
0812 14:52:29.232856 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:29.232858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:29.232864 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:52:29.232905 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:29.232906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:29.232914 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:29.232951 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:29.233037 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.233489 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.234134 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:29.234312 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:29.234385 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:29.234459 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:29.234484 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:29.234530 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:29.234542 (+ 12us) server_negotiation.cc:299] Negotiation successful
0812 14:52:29.234556 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:29.236387 18126 raft_consensus.cc:1261] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:29.236470 18629 raft_consensus.cc:1261] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Refusing update from remote peer 19b5b4341f48483cab4ffe423d82a1d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:29.236616 18788 consensus_queue.cc:1035] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1658c8617d0f46d8b1f78584bd14e13b" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:29.236714 18769 consensus_queue.cc:1035] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf240d905b654134a1c1a790149ff421" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:29.245754 17827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:29.245832 17827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:29.246366 17827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc10050a7e5a4477bcbf16b0c5581754: Sending DeleteTablet for 3 replicas of tablet 2784542f635346999bbe9acfc20751fb
I20260812 14:52:29.246629 18090 tablet_service.cc:1514] Processing DeleteTablet for tablet 2784542f635346999bbe9acfc20751fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:29 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260812 14:52:29.246634 18335 tablet_service.cc:1514] Processing DeleteTablet for tablet 2784542f635346999bbe9acfc20751fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:29 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42094
I20260812 14:52:29.246730 18801 tablet_replica.cc:330] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: stopping tablet replica
I20260812 14:52:29.246760 18589 tablet_service.cc:1514] Processing DeleteTablet for tablet 2784542f635346999bbe9acfc20751fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:29 UTC) from {username='oryx', principal='oryx/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55654
I20260812 14:52:29.246791 18801 raft_consensus.cc:2229] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:29.246842 18801 raft_consensus.cc:2258] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:29.246845 18799 tablet_replica.cc:330] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: stopping tablet replica
I20260812 14:52:29.246884 18799 raft_consensus.cc:2229] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:29.246922 18799 raft_consensus.cc:2258] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:29.246915 18798 tablet_replica.cc:330] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: stopping tablet replica
I20260812 14:52:29.246965 18798 raft_consensus.cc:2229] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:29.247002 18798 raft_consensus.cc:2258] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:29.247226 18799 ts_tablet_manager.cc:1905] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:29.247371 18798 ts_tablet_manager.cc:1905] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:29.248322 18801 ts_tablet_manager.cc:1905] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:29.248466 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 17974
I20260812 14:52:29.248553 18798 ts_tablet_manager.cc:1918] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:29.248606 18798 log.cc:1196] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/2784542f635346999bbe9acfc20751fb
I20260812 14:52:29.248790 18798 ts_tablet_manager.cc:1939] T 2784542f635346999bbe9acfc20751fb P 1658c8617d0f46d8b1f78584bd14e13b: Deleting consensus metadata
I20260812 14:52:29.249725 18801 ts_tablet_manager.cc:1918] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:29.249780 18801 log.cc:1196] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/2784542f635346999bbe9acfc20751fb
I20260812 14:52:29.249990 18801 ts_tablet_manager.cc:1939] T 2784542f635346999bbe9acfc20751fb P 19b5b4341f48483cab4ffe423d82a1d3: Deleting consensus metadata
I20260812 14:52:29.255620 17804 catalog_manager.cc:4841] TS 19b5b4341f48483cab4ffe423d82a1d3 (127.4.27.2:33607): tablet 2784542f635346999bbe9acfc20751fb (table test-table [id=b1cc29d89e87488ab87a1fdb471629cf]) successfully deleted
I20260812 14:52:29.257130 18799 ts_tablet_manager.cc:1918] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 14:52:29.257179 18799 log.cc:1196] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/2784542f635346999bbe9acfc20751fb
I20260812 14:52:29.257319 18799 ts_tablet_manager.cc:1939] T 2784542f635346999bbe9acfc20751fb P bf240d905b654134a1c1a790149ff421: Deleting consensus metadata
I20260812 14:52:29.257606 17803 catalog_manager.cc:4841] TS bf240d905b654134a1c1a790149ff421 (127.4.27.3:34295): tablet 2784542f635346999bbe9acfc20751fb (table test-table [id=b1cc29d89e87488ab87a1fdb471629cf]) successfully deleted
W20260812 14:52:29.262209 17804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:32891 (error 108)
I20260812 14:52:29.262725 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 18236
I20260812 14:52:29.262756 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:29.262579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:32891 (local address 127.0.0.1:55324)
0812 14:52:29.262674 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:29.262722 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
W20260812 14:52:29.262878 17804 catalog_manager.cc:4584] TS 1658c8617d0f46d8b1f78584bd14e13b (127.4.27.1:32891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2784542f635346999bbe9acfc20751fb: Network error: Client connection negotiation failed: client connection to 127.4.27.1:32891: connect: Connection refused (error 111)
I20260812 14:52:29.274252 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 18474
I20260812 14:52:29.284822 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 17774
2026-08-12T14:52:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2739 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:29 dist-test-slave-6zbq krb5kdc[18902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:29 dist-test-slave-6zbq krb5kdc[18902](info): set up 2 sockets
Aug 12 14:52:29 dist-test-slave-6zbq krb5kdc[18902](info): commencing operation
krb5kdc: starting...
W20260812 14:52:30.460783 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:30 dist-test-slave-6zbq krb5kdc[18902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:30.567071 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:34929
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:34929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:30.646960 18920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:30.647136 18920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:30.647157 18920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:30.648216 18920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:30.648248 18920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:30.648260 18920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:30.648272 18920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:30.648285 18920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:30.649678 18920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:34929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:34929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:30.649849 18920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:30.650007 18920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:30.654318 18931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:30.654328 18928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:30.654328 18927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:30.654452 18920 server_base.cc:1034] running on GCE node
I20260812 14:52:30.654721 18920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:30.655006 18920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:30.656143 18920 hybrid_clock.cc:648] HybridClock initialized: now 1786546350656131 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T14:52:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1372 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:30 dist-test-slave-6zbq krb5kdc[18938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:30 dist-test-slave-6zbq krb5kdc[18938](info): set up 2 sockets
Aug 12 14:52:30 dist-test-slave-6zbq krb5kdc[18938](info): commencing operation
krb5kdc: starting...
W20260812 14:52:31.830163 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:31 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:31.943318 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36551
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:32.023211 18976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:32.023402 18976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:32.023422 18976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:32.024482 18976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:32.024514 18976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:32.024526 18976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:32.024537 18976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:32.024547 18976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:32.025935 18976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:32.026120 18976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:32.026283 18976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:32.030716 18984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.030853 18976 server_base.cc:1034] running on GCE node
W20260812 14:52:32.030812 18987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.030768 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.031118 18976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:32.031395 18976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:32.032543 18976 hybrid_clock.cc:648] HybridClock initialized: now 1786546352032525 us; error 30 us; skew 500 ppm
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:32.035014 18976 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:32.035398 18976 webserver.cc:460] Webserver started at http://127.4.27.62:39147/ using document root <none> and password file <none>
I20260812 14:52:32.035580 18976 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:32.035619 18976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:32.035705 18976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:32.036679 18976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "de068b8b67a14927a646faeed77d974a"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "cfb15ea9b22bc68afa10e17687a35c35"
server_key_iv: "386ed89d8329f05b25de4ce3a0e30b30"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.036974 18976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "de068b8b67a14927a646faeed77d974a"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "cfb15ea9b22bc68afa10e17687a35c35"
server_key_iv: "386ed89d8329f05b25de4ce3a0e30b30"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.038156 18976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:32.038833 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.038973 18976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.039055 18976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "de068b8b67a14927a646faeed77d974a"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "cfb15ea9b22bc68afa10e17687a35c35"
server_key_iv: "386ed89d8329f05b25de4ce3a0e30b30"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.039122 18976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:32.058384 18976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:32.060168 18976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:32.060283 18976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:32.063951 18976 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:36551
I20260812 14:52:32.064014 19102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:36551 every 8 connection(s)
I20260812 14:52:32.064301 18976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:32.065097 19103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:32.067291 19103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Bootstrap starting.
I20260812 14:52:32.067734 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 18976
I20260812 14:52:32.067860 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:32.067900 19103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:32.068061 4204 external_mini_cluster.cc:1428] Setting key e59b74839801eca0d03acb5cad89761f
I20260812 14:52:32.068235 19103 log.cc:824] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:32.068872 19103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: No bootstrap required, opened a new log
I20260812 14:52:32.070938 19103 raft_consensus.cc:348] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.071076 19103 raft_consensus.cc:374] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:32.071110 19103 raft_consensus.cc:729] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de068b8b67a14927a646faeed77d974a, State: Initialized, Role: FOLLOWER
I20260812 14:52:32.071241 19103 consensus_queue.cc:260] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.071314 19103 raft_consensus.cc:388] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:32.071357 19103 raft_consensus.cc:482] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:32.071403 19103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:32.071919 19103 raft_consensus.cc:504] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.072014 19103 leader_election.cc:302] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de068b8b67a14927a646faeed77d974a; no voters:
I20260812 14:52:32.072173 19103 leader_election.cc:288] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:32.072212 19115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:32.072340 19115 raft_consensus.cc:686] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 LEADER]: Becoming Leader. State: Replica: de068b8b67a14927a646faeed77d974a, State: Running, Role: LEADER
I20260812 14:52:32.072429 19103 sys_catalog.cc:564] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:32.072490 19115 consensus_queue.cc:237] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.072901 19118 sys_catalog.cc:455] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de068b8b67a14927a646faeed77d974a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } } }
I20260812 14:52:32.073006 19118 sys_catalog.cc:458] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:32.072898 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de068b8b67a14927a646faeed77d974a. Latest consensus state: current_term: 1 leader_uuid: "de068b8b67a14927a646faeed77d974a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } } }
I20260812 14:52:32.073158 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:32.073459 19120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:32.073942 19120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:32.075592 19120 catalog_manager.cc:1347] Generated new cluster ID: 193578dc14aa4653980800bc75851dd5
I20260812 14:52:32.075640 19120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:32.078976 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.068507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.4.27.62:36551)
0812 14:52:32.068653 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.068662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.068695 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:52:32.069163 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.069171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.069219 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.069346 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.069979 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.070887 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.071755 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.072199 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.072296 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.075414 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:32.075418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:32.075418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:32.075429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:32.077401 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.077757 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.077760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.077760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.077875 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.078050 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.078051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.078051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.078388 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:32.078543 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:32.078630 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:52:32.078766 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260812 14:52:32.085544 19120 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:32.086232 19120 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:32.098354 19120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Generated new TSK 0
I20260812 14:52:32.098516 19120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:32.126675 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36551
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:32.206413 19147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:32.206601 19147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:32.206622 19147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:32.207746 19147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:32.207785 19147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:32.207811 19147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:32.209182 19147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:32.209379 19147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:32.209583 19147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:32.214164 19162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.214143 19156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.214166 19155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.214226 19147 server_base.cc:1034] running on GCE node
I20260812 14:52:32.214591 19147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:32.214821 19147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:32.215955 19147 hybrid_clock.cc:648] HybridClock initialized: now 1786546352215940 us; error 28 us; skew 500 ppm
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:32.218194 19147 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:32.218617 19147 webserver.cc:460] Webserver started at http://127.4.27.1:39787/ using document root <none> and password file <none>
I20260812 14:52:32.218806 19147 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:32.218852 19147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:32.218969 19147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:32.219908 19147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "ac6031e03af6499b9905adbfd10214df"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "02f12de8ceb333c099f10d6989ec30d7"
server_key_iv: "c0028b0701162331f835b5c83e1f187c"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.220220 19147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "ac6031e03af6499b9905adbfd10214df"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "02f12de8ceb333c099f10d6989ec30d7"
server_key_iv: "c0028b0701162331f835b5c83e1f187c"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.221510 19147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:32.222270 19171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.222440 19147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:32.222525 19147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "ac6031e03af6499b9905adbfd10214df"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "02f12de8ceb333c099f10d6989ec30d7"
server_key_iv: "c0028b0701162331f835b5c83e1f187c"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.222591 19147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:32.236428 19147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:32.238225 19147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:32.238420 19147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:32.238626 19147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:32.238907 19147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:32.238940 19147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.238968 19147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:32.238991 19147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.244371 19147 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37843
I20260812 14:52:32.244406 19363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37843 every 8 connection(s)
I20260812 14:52:32.244674 19147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:32.251257 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 19147
I20260812 14:52:32.251351 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:32.251545 4204 external_mini_cluster.cc:1428] Setting key 28db07c2e49919eab3db2743a3c61afd
I20260812 14:52:32.252736 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.245307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41877 (local address 127.4.27.62:36551)
0812 14:52:32.245380 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.245382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.245516 (+ 134us) server_negotiation.cc:407] Connection header received
0812 14:52:32.245943 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.245944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.245954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.245997 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.246852 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.247335 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.248211 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.248396 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.248439 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.250212 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:32.250214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:32.250214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:32.250221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:32.251283 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.251811 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.251812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.251813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.251873 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.252118 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.252118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.252119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.252253 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:32.252311 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:32.252598 (+ 287us) server_negotiation.cc:299] Negotiation successful
0812 14:52:32.252665 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 14:52:32.252923 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.245281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:41877)
0812 14:52:32.245414 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:32.245447 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:32.245873 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:32.246052 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:32.246055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:32.246603 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:32.246780 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.246785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.247392 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.247392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.248150 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.248152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.248447 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.248448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.248535 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.248879 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:32.248887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:32.250119 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:32.251365 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:32.251372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:32.251374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:32.251652 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:32.251941 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:32.251941 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:32.251941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:32.252059 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:32.252316 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:32.252323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:32.252476 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 14:52:32.252574 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 14:52:32.252725 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260812 14:52:32.253324 19366 heartbeater.cc:347] Connected to a master server at 127.4.27.62:36551
I20260812 14:52:32.253410 19366 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:32.253561 19366 heartbeater.cc:511] Master 127.4.27.62:36551 requested a full tablet report, sending...
I20260812 14:52:32.254070 19035 ts_manager.cc:194] Registered new tserver with Master: ac6031e03af6499b9905adbfd10214df (127.4.27.1:37843)
I20260812 14:52:32.254927 19035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:41877
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:32.274048 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36551
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:32.351845 19378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:32.352027 19378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:32.352048 19378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:32.353137 19378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:32.353176 19378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:32.353202 19378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:32.354897 19378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:32.355130 19378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:32.355324 19378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:32.359926 19390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.360085 19378 server_base.cc:1034] running on GCE node
W20260812 14:52:32.359947 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.359936 19387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.360344 19378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:32.360558 19378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:32.361691 19378 hybrid_clock.cc:648] HybridClock initialized: now 1786546352361682 us; error 28 us; skew 500 ppm
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:32.364037 19378 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:32.364559 19378 webserver.cc:460] Webserver started at http://127.4.27.2:41207/ using document root <none> and password file <none>
I20260812 14:52:32.364797 19378 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:32.364842 19378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:32.364929 19378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:32.365849 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "c0833771f8004808a44dabb0adbe0d34"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "2bd384835fb82bf469a9ac4e58ab6792"
server_key_iv: "bc7aad14a41321a77e90384292f65d44"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.366125 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "c0833771f8004808a44dabb0adbe0d34"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "2bd384835fb82bf469a9ac4e58ab6792"
server_key_iv: "bc7aad14a41321a77e90384292f65d44"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.367314 19378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:32.367935 19404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.368067 19378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:32.368146 19378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "c0833771f8004808a44dabb0adbe0d34"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "2bd384835fb82bf469a9ac4e58ab6792"
server_key_iv: "bc7aad14a41321a77e90384292f65d44"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.368211 19378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:32.386039 19378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:32.387954 19378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:32.388077 19378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:32.388271 19378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:32.388522 19378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:32.388553 19378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.388578 19378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:32.388597 19378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.393929 19378 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:34489
I20260812 14:52:32.393966 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:34489 every 8 connection(s)
I20260812 14:52:32.394240 19378 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:32.397951 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 19378
I20260812 14:52:32.398057 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:32.398269 4204 external_mini_cluster.cc:1428] Setting key 01f9aea9759201de4383866472814db8
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:32.402587 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.394850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:55785 (local address 127.4.27.62:36551)
0812 14:52:32.394927 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.394929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.395054 (+ 125us) server_negotiation.cc:407] Connection header received
0812 14:52:32.395458 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.395459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.395469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.395513 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.396284 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.396759 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.397607 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.397803 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.397867 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.400016 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:32.400017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:32.400017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:32.400023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:32.401407 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.401767 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.401768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.401768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.401827 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.402021 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.402022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.402022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.402144 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:32.402208 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:32.402421 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:52:32.402506 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:52:32.402634 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.394811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:55785)
0812 14:52:32.394952 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:32.394982 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:32.395387 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:32.395561 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:32.395564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:32.396049 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:32.396223 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.396227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.396815 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.396816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.397551 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.397552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.397868 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.397869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.397953 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.398410 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:32.398419 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:32.399932 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:32.401481 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:32.401487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:32.401489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:32.401689 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:32.401885 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:32.401885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:32.401886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:32.401970 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:32.402197 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:32.402201 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:32.402299 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 14:52:32.402387 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:52:32.402494 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260812 14:52:32.403050 19616 heartbeater.cc:347] Connected to a master server at 127.4.27.62:36551
I20260812 14:52:32.403134 19616 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:32.403313 19616 heartbeater.cc:511] Master 127.4.27.62:36551 requested a full tablet report, sending...
I20260812 14:52:32.403683 19035 ts_manager.cc:194] Registered new tserver with Master: c0833771f8004808a44dabb0adbe0d34 (127.4.27.2:34489)
I20260812 14:52:32.404155 19035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:55785
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:32.422147 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36551
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:32.502362 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:32.502550 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:32.502573 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:32.503665 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:32.503698 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:32.503722 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:32.505093 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:36551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:32.505272 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:32.505445 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:32.510174 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.510174 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.510185 19636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.510371 19629 server_base.cc:1034] running on GCE node
I20260812 14:52:32.510615 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:32.510835 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:32.511996 19629 hybrid_clock.cc:648] HybridClock initialized: now 1786546352511984 us; error 26 us; skew 500 ppm
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:32.514288 19629 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:32.514699 19629 webserver.cc:460] Webserver started at http://127.4.27.3:34843/ using document root <none> and password file <none>
I20260812 14:52:32.514889 19629 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:32.514935 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:32.515053 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:32.516010 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "23dda6bf8a73488f9fdc7026d0c164aa"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "f010bbb9f960e5c8e9105aaf62cec9ce"
server_key_iv: "b64739407318e240e1b97eb20aee343f"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.516328 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "23dda6bf8a73488f9fdc7026d0c164aa"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "f010bbb9f960e5c8e9105aaf62cec9ce"
server_key_iv: "b64739407318e240e1b97eb20aee343f"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.517494 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:32.518129 19655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.518415 19629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:32.518512 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "23dda6bf8a73488f9fdc7026d0c164aa"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "f010bbb9f960e5c8e9105aaf62cec9ce"
server_key_iv: "b64739407318e240e1b97eb20aee343f"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.518584 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:32.531930 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:32.533475 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:32.533597 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:32.533780 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:32.534041 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:32.534073 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.534099 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:32.534133 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.539494 19629 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:41599
I20260812 14:52:32.539531 19875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:41599 every 8 connection(s)
I20260812 14:52:32.539798 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:32.546329 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 19629
I20260812 14:52:32.546460 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:32.546651 4204 external_mini_cluster.cc:1428] Setting key da3a9193d34acfe2c33a708548e4e3e4
I20260812 14:52:32.548075 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.540390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:47205 (local address 127.4.27.62:36551)
0812 14:52:32.540466 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.540468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.540600 (+ 132us) server_negotiation.cc:407] Connection header received
0812 14:52:32.541025 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.541026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.541040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.541101 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.541983 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.542580 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.543505 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.543694 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.543749 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.545588 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:32.545590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:32.545590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:32.545597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:32.546813 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.547283 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.547284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.547285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.547349 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:32.547596 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:32.547596 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:32.547597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:32.547708 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:32.547757 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:32.547956 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 14:52:32.548015 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:52:32.548257 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.540358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:47205)
0812 14:52:32.540496 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:32.540526 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:32.540939 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:32.541166 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:32.541170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:32.541702 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:32.541899 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.541904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.542666 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.542667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.543426 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.543428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.543760 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.543761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.543868 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.544281 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:32.544288 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:32.545505 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:32.546910 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:32.546917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:32.546919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:32.547216 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:32.547412 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:32.547413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:32.547413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:32.547543 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:32.547755 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:32.547758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:32.547865 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 14:52:32.547942 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 14:52:32.548054 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260812 14:52:32.548641 19876 heartbeater.cc:347] Connected to a master server at 127.4.27.62:36551
I20260812 14:52:32.548729 19876 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:32.548861 19876 heartbeater.cc:511] Master 127.4.27.62:36551 requested a full tablet report, sending...
I20260812 14:52:32.549165 19035 ts_manager.cc:194] Registered new tserver with Master: 23dda6bf8a73488f9fdc7026d0c164aa (127.4.27.3:41599)
I20260812 14:52:32.549496 19035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:47205
I20260812 14:52:32.558099 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab.
Entry for principal oryx/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab.
WARNING: no policy specified for oryx/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab.
Entry for principal oryx/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab.
WARNING: no policy specified for oryx/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab.
Entry for principal oryx/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab.
I20260812 14:52:32.618476 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 19147
I20260812 14:52:32.627539 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 19378
I20260812 14:52:32.640739 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 19629
I20260812 14:52:32.654603 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 18976
I20260812 14:52:32.674238 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36551
--webserver_interface=127.4.27.62
--webserver_port=39147
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:32.753198 19893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:32.753396 19893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:32.753417 19893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:32.754515 19893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:32.754549 19893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:32.754563 19893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:32.754575 19893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:32.754585 19893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:32.755965 19893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=39147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:32.756143 19893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:32.756313 19893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:32.760722 19903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.760730 19902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.760766 19906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.760910 19893 server_base.cc:1034] running on GCE node
I20260812 14:52:32.761145 19893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:32.761407 19893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:32.762543 19893 hybrid_clock.cc:648] HybridClock initialized: now 1786546352762529 us; error 31 us; skew 500 ppm
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:32.764859 19893 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:32.765240 19893 webserver.cc:460] Webserver started at http://127.4.27.62:39147/ using document root <none> and password file <none>
I20260812 14:52:32.765422 19893 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:32.765475 19893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:32.766749 19893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:32.767323 19920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.767464 19893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:32.767544 19893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "de068b8b67a14927a646faeed77d974a"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "cfb15ea9b22bc68afa10e17687a35c35"
server_key_iv: "386ed89d8329f05b25de4ce3a0e30b30"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.767855 19893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:32.778719 19893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:32.780655 19893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:32.780786 19893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:32.784504 19893 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:36551
I20260812 14:52:32.784591 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:36551 every 8 connection(s)
I20260812 14:52:32.784902 19893 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:32.786324 20022 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:52:32.786981 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Bootstrap starting.
I20260812 14:52:32.788617 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 19893
I20260812 14:52:32.788875 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:37843
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:36551
--webserver_port=39787
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:52:32.795339 20022 log.cc:824] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:32.797333 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:52:32.797608 20022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Bootstrap complete.
I20260812 14:52:32.800033 20022 raft_consensus.cc:348] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.800261 20022 raft_consensus.cc:729] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de068b8b67a14927a646faeed77d974a, State: Initialized, Role: FOLLOWER
I20260812 14:52:32.800421 20022 consensus_queue.cc:260] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.800501 20022 raft_consensus.cc:388] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:32.800552 20022 raft_consensus.cc:482] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:32.800599 20022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:52:32.801586 20022 raft_consensus.cc:504] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.801683 20022 leader_election.cc:302] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de068b8b67a14927a646faeed77d974a; no voters:
I20260812 14:52:32.801848 20022 leader_election.cc:288] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:52:32.801894 20033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:52:32.802060 20033 raft_consensus.cc:686] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [term 2 LEADER]: Becoming Leader. State: Replica: de068b8b67a14927a646faeed77d974a, State: Running, Role: LEADER
I20260812 14:52:32.802124 20022 sys_catalog.cc:564] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:32.802767 20033 consensus_queue.cc:237] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } }
I20260812 14:52:32.803287 20038 sys_catalog.cc:455] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de068b8b67a14927a646faeed77d974a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } } }
I20260812 14:52:32.803375 20038 sys_catalog.cc:458] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:32.803574 20056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:32.803576 20037 sys_catalog.cc:455] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de068b8b67a14927a646faeed77d974a. Latest consensus state: current_term: 2 leader_uuid: "de068b8b67a14927a646faeed77d974a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de068b8b67a14927a646faeed77d974a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36551 } } }
I20260812 14:52:32.803843 20037 sys_catalog.cc:458] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:32.804211 20056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:32.804337 20056 catalog_manager.cc:1259] Loaded cluster ID: 193578dc14aa4653980800bc75851dd5
I20260812 14:52:32.804369 20056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:32.805135 20056 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:32.805290 20056 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de068b8b67a14927a646faeed77d974a: Loaded TSK: 0
I20260812 14:52:32.805485 20056 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:52:32.888033 20029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:32.888234 20029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:32.888258 20029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:32.889403 20029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:32.889442 20029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:32.889468 20029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:32.890914 20029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.1.keytab
--principal=oryx/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:37843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=39787
--tserver_master_addrs=127.4.27.62:36551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:32.891117 20029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:32.891320 20029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:32.896422 20066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.896430 20065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:32.896762 20074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:32.896929 20029 server_base.cc:1034] running on GCE node
I20260812 14:52:32.897181 20029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:32.897468 20029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:32.898643 20029 hybrid_clock.cc:648] HybridClock initialized: now 1786546352898617 us; error 39 us; skew 500 ppm
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546352, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:32.901630 20029 init.cc:377] Logged in from keytab as oryx/127.4.27.1@KRBTEST.COM (short username oryx)
I20260812 14:52:32.902174 20029 webserver.cc:460] Webserver started at http://127.4.27.1:39787/ using document root <none> and password file <none>
I20260812 14:52:32.902446 20029 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:32.902531 20029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:32.904254 20029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:52:32.904901 20084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.905081 20029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:32.905179 20029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "ac6031e03af6499b9905adbfd10214df"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "02f12de8ceb333c099f10d6989ec30d7"
server_key_iv: "c0028b0701162331f835b5c83e1f187c"
server_key_version: "encryptionkey@0"
I20260812 14:52:32.905493 20029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:32.924870 20029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:32.926854 20029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:32.926960 20029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:32.927181 20029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:32.927474 20029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:32.927506 20029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.927536 20029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:32.927559 20029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:32.933076 20029 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37843
I20260812 14:52:32.933131 20281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37843 every 8 connection(s)
I20260812 14:52:32.933466 20029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:32.934759 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 20029
I20260812 14:52:32.935062 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:34489
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:36551
--webserver_port=41207
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:32.942857 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.934150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:43953)
0812 14:52:32.934307 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:32.934365 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:32.934901 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:32.935430 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:32.935433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:32.935962 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:32.936147 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.936151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.937581 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.937583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.938783 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.938786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.939354 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.939355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.939492 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.939962 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:32.939972 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:32.942647 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260812 14:52:32.942968 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:32.934215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:43953 (local address 127.4.27.62:36551)
0812 14:52:32.934445 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.934453 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.934491 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:52:32.935099 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.935104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.935172 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.935298 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.936261 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.937469 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.938870 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.939264 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.939342 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.942796 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:43953: BlockingRecv error: recv got EOF from 127.4.27.1:43953 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260812 14:52:32.943600 20282 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:36551 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.27.62:36551: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:32.949784 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.943837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:41293)
0812 14:52:32.944010 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:32.944021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:32.944072 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:32.944224 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:32.944225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:32.944364 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:32.944421 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.944423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.945276 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.945277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.946024 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.946027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.946491 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.946492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.946560 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.946945 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:32.946949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:32.949727 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20260812 14:52:32.949880 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:32.943899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41293 (local address 127.4.27.62:36551)
0812 14:52:32.943948 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.943949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.944069 (+ 120us) server_negotiation.cc:407] Connection header received
0812 14:52:32.944142 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.944143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.944154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.944212 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.944485 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.945201 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.946119 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.946422 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.946480 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.949824 (+ 3344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:41293: BlockingRecv error: recv got EOF from 127.4.27.1:41293 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:32 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:32.959944 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:32.950201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:41169)
0812 14:52:32.950242 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:32.950251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:32.950281 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:32.950416 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:32.950417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:32.950551 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:32.950583 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.950584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.951357 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.951358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.952145 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:32.952147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.952473 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.952473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:32.952522 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.958269 (+ 5747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:32.958275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:32.959884 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 14:52:32.960008 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:32.950251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41169 (local address 127.4.27.62:36551)
0812 14:52:32.950282 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:32.950284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:32.950291 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:32.950328 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:32.950329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:32.950354 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:32.950405 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:32.950648 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.951281 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.952232 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:32.952407 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:32.952466 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:32.959957 (+ 7491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:41169: BlockingRecv error: recv got EOF from 127.4.27.1:41169 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:52:32.960049 20282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:52:33.019268 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:33.019449 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:33.019470 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:33.020552 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:33.020588 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:33.020612 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:33.022009 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.2.keytab
--principal=oryx/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:34489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=41207
--tserver_master_addrs=127.4.27.62:36551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:33.022184 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:33.022372 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:33.026856 20299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:33.026880 20309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:33.026969 20289 server_base.cc:1034] running on GCE node
W20260812 14:52:33.026902 20306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:33.027187 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:33.027382 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:33.028506 20289 hybrid_clock.cc:648] HybridClock initialized: now 1786546353028493 us; error 28 us; skew 500 ppm
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546353, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:33.030809 20289 init.cc:377] Logged in from keytab as oryx/127.4.27.2@KRBTEST.COM (short username oryx)
I20260812 14:52:33.031178 20289 webserver.cc:460] Webserver started at http://127.4.27.2:41207/ using document root <none> and password file <none>
I20260812 14:52:33.031390 20289 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:33.031453 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:33.032657 20289 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:33.033134 20325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.033278 20289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.033347 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "c0833771f8004808a44dabb0adbe0d34"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "2bd384835fb82bf469a9ac4e58ab6792"
server_key_iv: "bc7aad14a41321a77e90384292f65d44"
server_key_version: "encryptionkey@0"
I20260812 14:52:33.033644 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:33.044113 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:33.045961 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:33.046082 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:33.046267 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:33.046540 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:33.046573 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.046598 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:33.046623 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.051869 20289 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:34489
I20260812 14:52:33.051944 20538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:34489 every 8 connection(s)
I20260812 14:52:33.052199 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:52:33.058419 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.052830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:57543 (local address 127.4.27.62:36551)
0812 14:52:33.052867 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.052869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.053077 (+ 208us) server_negotiation.cc:407] Connection header received
0812 14:52:33.053472 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.053474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.053486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.053533 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.054333 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.054828 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.055711 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.055887 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.055929 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.058367 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:57543: BlockingRecv error: recv got EOF from 127.4.27.2:57543 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:52:33.058427 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.052831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:57543)
0812 14:52:33.052964 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.052996 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.053401 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.053584 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.053586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.054091 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.054270 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.054274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.054891 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.054892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.055650 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.055652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.055938 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.055939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.056030 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.056374 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.056381 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.058266 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
W20260812 14:52:33.058687 20539 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:36551 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.27.62:36551: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:33.061326 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 20289
I20260812 14:52:33.061590 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:41599
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:36551
--webserver_port=34843
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:33.062194 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.058857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:39583)
0812 14:52:33.058899 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.058909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.058949 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.059030 (+ 81us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.059031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.059130 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.059170 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.059170 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.059674 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.059675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.060136 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.060137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.060392 (+ 255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.060392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.060435 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.060713 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.060717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.062152 (+ 1435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:52:33.062300 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.058909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:39583 (local address 127.4.27.62:36551)
0812 14:52:33.058949 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.058950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.058958 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:33.058980 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.058980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.058987 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.059026 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.059215 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.059627 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.060188 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.060348 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.060387 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.062250 (+ 1863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:39583: BlockingRecv error: recv got EOF from 127.4.27.2:39583 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:33.066403 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.062414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:51227)
0812 14:52:33.062448 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.062456 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.062484 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.062603 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.062603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.062705 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.062729 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.062730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.063449 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.063449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.064115 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.064117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.064432 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.064433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.064478 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.064775 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.064779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.066329 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:52:33.066489 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.062455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:51227 (local address 127.4.27.62:36551)
0812 14:52:33.062481 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.062483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.062490 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:33.062533 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.062534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.062542 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.062601 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.062784 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.063377 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.064211 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.064374 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.064414 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.066440 (+ 2026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:51227: BlockingRecv error: recv got EOF from 127.4.27.2:51227 (error 108)
Metrics: {"server-negotiator.queue_time_us":11}
W20260812 14:52:33.066499 20539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:52:33.141562 20542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:33.141772 20542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:33.141794 20542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:33.142959 20542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:33.142999 20542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:33.143023 20542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:33.144408 20542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/krb5kdc/oryx_127.4.27.3.keytab
--principal=oryx/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:41599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=34843
--tserver_master_addrs=127.4.27.62:36551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:33.144610 20542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:33.144821 20542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:33.149613 20549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:33.149614 20556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:33.149789 20542 server_base.cc:1034] running on GCE node
W20260812 14:52:33.149670 20550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:33.150202 20542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:33.150609 20542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:33.151778 20542 hybrid_clock.cc:648] HybridClock initialized: now 1786546353151757 us; error 36 us; skew 500 ppm
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546353, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:33.154071 20542 init.cc:377] Logged in from keytab as oryx/127.4.27.3@KRBTEST.COM (short username oryx)
I20260812 14:52:33.154511 20542 webserver.cc:460] Webserver started at http://127.4.27.3:34843/ using document root <none> and password file <none>
I20260812 14:52:33.154685 20542 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:33.154735 20542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:33.155917 20542 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:52:33.156384 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.156536 20542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.156617 20542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "23dda6bf8a73488f9fdc7026d0c164aa"
format_stamp: "Formatted at 2026-08-12 14:52:32 on dist-test-slave-6zbq"
server_key: "f010bbb9f960e5c8e9105aaf62cec9ce"
server_key_iv: "b64739407318e240e1b97eb20aee343f"
server_key_version: "encryptionkey@0"
I20260812 14:52:33.156956 20542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:33.171481 20542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:33.172979 20542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:33.173077 20542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:33.173237 20542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:33.173458 20542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:33.173487 20542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.173506 20542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:33.173522 20542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:33.178694 20542 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:41599
I20260812 14:52:33.178756 20779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:41599 every 8 connection(s)
I20260812 14:52:33.179026 20542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 14:52:33.185204 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.179627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:46565 (local address 127.4.27.62:36551)
0812 14:52:33.179694 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.179696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.179881 (+ 185us) server_negotiation.cc:407] Connection header received
0812 14:52:33.180306 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.180307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.180315 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.180363 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.181176 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.181640 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.182526 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.182704 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.182752 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.185159 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:46565: BlockingRecv error: recv got EOF from 127.4.27.3:46565 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 14:52:33.185207 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.179611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:46565)
0812 14:52:33.179765 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.179804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.180226 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.180418 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.180421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.180928 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.181112 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.181116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.181697 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.181698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.182464 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.182467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.182757 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.182758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.182846 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.183202 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.183210 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.185065 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260812 14:52:33.185384 20781 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:36551 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.27.62:36551: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
I20260812 14:52:33.187244 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 20542
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:33.189662 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.185567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:41391)
0812 14:52:33.185608 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.185617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.185651 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.185846 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.185846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.185962 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.186000 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.186001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.186667 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.186668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.187146 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.187147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.187512 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.187514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.187578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.187998 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.188003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.189621 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:52:33.189795 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.185703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:41391 (local address 127.4.27.62:36551)
0812 14:52:33.185752 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.185753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.185760 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:33.185785 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.185785 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.185793 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.185835 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.186063 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.186603 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.187215 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.187445 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.187510 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.189722 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:41391: BlockingRecv error: recv got EOF from 127.4.27.3:41391 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:33.193658 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.190049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:50961)
0812 14:52:33.190144 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.190155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.190196 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.190317 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.190318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.190471 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.190495 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.190496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.191043 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.191044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.191513 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.191514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.191807 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.191808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.191855 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.192157 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.192160 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.193606 (+ 1446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 14:52:33.193732 20287 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.190104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:50961 (local address 127.4.27.62:36551)
0812 14:52:33.190150 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.190151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.190184 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:52:33.190249 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.190250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.190257 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.190308 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.190557 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.190979 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.191578 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.191751 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.191802 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.193683 (+ 1881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:50961: BlockingRecv error: recv got EOF from 127.4.27.3:50961 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260812 14:52:33.193773 20781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:52:33.194245 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.187586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.4.27.62:36551)
0812 14:52:33.187896 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.187897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.187902 (+ 5us) server_negotiation.cc:407] Connection header received
0812 14:52:33.187918 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.187918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.187926 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.187983 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.188577 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.189060 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.190104 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.190302 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.190381 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.191454 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:33.191458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:33.191459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:33.191470 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:33.192982 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:33.193283 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:33.193287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:33.193288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:33.193408 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:33.193591 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:33.193592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:33.193592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:33.193863 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:33.193994 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:33.194079 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:52:33.194161 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":71,"threads_started":1}
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:33 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:33.964826 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:33.960410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:60581)
0812 14:52:33.960571 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:33.960583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:33.960646 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:33.960778 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:33.960779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:33.960935 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:33.960988 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.960989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.961762 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.961763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.962318 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:33.962320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.962693 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.962694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:33.962740 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.963145 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:33.963151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:33.964759 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260812 14:52:33.964912 20789 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:33.960510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:60581 (local address 127.4.27.62:36551)
0812 14:52:33.960617 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:33.960619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:33.960630 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:52:33.960706 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:33.960707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:33.960718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:33.960764 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:33.961078 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.961663 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.962418 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:33.962624 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:33.962679 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:33.964847 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:60581: BlockingRecv error: recv got EOF from 127.4.27.1:60581 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
Aug 12 14:52:34 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:34 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:34.070917 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:34.066790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:55179)
0812 14:52:34.066920 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:34.066930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:34.066987 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:34.067095 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:34.067096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:34.067247 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:34.067299 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:34.067301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.067883 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.067884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:34.068413 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:34.068415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.068680 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.068680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:34.068715 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:34.069149 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:34.069154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:34.070834 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
W20260812 14:52:34.071010 20789 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:34.066843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:55179 (local address 127.4.27.62:36551)
0812 14:52:34.066890 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:34.066891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:34.066980 (+ 89us) server_negotiation.cc:407] Connection header received
0812 14:52:34.067027 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:34.067028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:34.067039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:34.067086 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:34.067349 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.067830 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.068470 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.068637 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.068677 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:34.070956 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:55179: BlockingRecv error: recv got EOF from 127.4.27.2:55179 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 14:52:34 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:34 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:34.198446 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:34.194064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:49031)
0812 14:52:34.194199 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:34.194212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:34.194269 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:34.194399 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:34.194400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:34.194550 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:34.194598 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:34.194599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.195176 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.195176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:34.195713 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:34.195715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.196070 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.196072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:34.196124 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:34.196602 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:34.196607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:34.198382 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
W20260812 14:52:34.198549 20789 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:34.194112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:49031 (local address 127.4.27.62:36551)
0812 14:52:34.194159 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:34.194162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:34.194264 (+ 102us) server_negotiation.cc:407] Connection header received
0812 14:52:34.194318 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:34.194319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:34.194331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:34.194385 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:34.194650 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.195125 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.195791 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.195982 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.196037 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:34.198483 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:49031: BlockingRecv error: recv got EOF from 127.4.27.3:49031 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 12 14:52:34 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:34 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:34.969600 20793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:34.965283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:51687)
0812 14:52:34.965414 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:34.965425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:34.965471 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:34.965603 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:34.965603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:34.965760 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:34.965812 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:34.965813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.966449 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.966449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:34.967095 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:34.967099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.967479 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.967480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:34.967523 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:34.967951 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:34.967955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:34.969544 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
W20260812 14:52:34.969708 20794 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:34.965354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:51687 (local address 127.4.27.62:36551)
0812 14:52:34.965465 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:34.965467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:34.965475 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:34.965530 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:34.965531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:34.965542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:34.965592 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:34.965864 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.966397 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.967168 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:34.967414 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:34.967461 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:34.969630 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:51687: BlockingRecv error: recv got EOF from 127.4.27.1:51687 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
Aug 12 14:52:35 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:35 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:35.075989 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:35.071396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:38751)
0812 14:52:35.071513 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:35.071524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:35.071580 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:35.071772 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:35.071774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:35.071944 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:35.071995 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:35.071996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.072720 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.072722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:35.073274 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:35.073276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.073736 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.073737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:35.073810 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:35.074288 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:35.074294 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:35.075932 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
W20260812 14:52:35.076097 20794 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:35.071470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:38751 (local address 127.4.27.62:36551)
0812 14:52:35.071544 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:35.071545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:35.071569 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:52:35.071641 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:35.071642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:35.071654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:35.071710 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:35.072096 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.072616 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.073376 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.073577 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.073638 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:35.076021 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:38751: BlockingRecv error: recv got EOF from 127.4.27.2:38751 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 14:52:35 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:35 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:35.203370 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:35.198863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:37899)
0812 14:52:35.198982 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:35.198994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:35.199052 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:35.199175 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:35.199176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:35.199345 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:35.199390 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:35.199391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.200017 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.200019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:35.200542 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:35.200544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.200873 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.200874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:35.200930 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:35.201373 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:35.201378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:35.203290 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
W20260812 14:52:35.203434 20794 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:35.198971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:37899 (local address 127.4.27.62:36551)
0812 14:52:35.199035 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:35.199037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:35.199046 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:35.199115 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:35.199116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:35.199129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:35.199170 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:35.199457 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.199945 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.200627 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.200817 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.200868 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:35.203385 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:37899: BlockingRecv error: recv got EOF from 127.4.27.3:37899 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 14:52:35 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:35 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:35.974574 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:35.970049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:35551)
0812 14:52:35.970182 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:35.970192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:35.970245 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:35.970402 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:35.970403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:35.970609 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:35.970674 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:35.970677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.971485 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.971485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:35.972032 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:35.972035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.972329 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.972330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:35.972364 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:35.972796 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:35.972802 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:35.974501 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
W20260812 14:52:35.974670 20801 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:35.970106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:35551 (local address 127.4.27.62:36551)
0812 14:52:35.970249 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:35.970251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:35.970261 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:35.970288 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:35.970289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:35.970299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:35.970397 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:35.970764 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.971422 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.972094 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:35.972284 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:35.972324 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:35.974597 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:35551: BlockingRecv error: recv got EOF from 127.4.27.1:35551 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
Aug 12 14:52:36 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:36 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:36.080744 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:36.076430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:42133)
0812 14:52:36.076556 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:36.076568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:36.076624 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:36.076744 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:36.076744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:36.076897 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:36.076948 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:36.076950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.077540 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.077541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:36.078082 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:36.078083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.078452 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.078453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:36.078492 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:36.078987 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:36.078993 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:36.080669 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
W20260812 14:52:36.080816 20801 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:36.076507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:42133 (local address 127.4.27.62:36551)
0812 14:52:36.076559 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:36.076562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:36.076615 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:52:36.076671 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:36.076671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:36.076683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:36.076733 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:36.076999 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.077486 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.078135 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.078316 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.078379 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:36.080755 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:42133: BlockingRecv error: recv got EOF from 127.4.27.2:42133 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 12 14:52:36 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:36 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:36.208333 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:36.203792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:52019)
0812 14:52:36.203926 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:36.203936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:36.203995 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:36.204109 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:36.204110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:36.204280 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:36.204328 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:36.204330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.204942 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.204942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:36.205471 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:36.205473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.205782 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.205783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:36.205817 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:36.206198 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:36.206201 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:36.208264 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
W20260812 14:52:36.208506 20801 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:36.203835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:52019 (local address 127.4.27.62:36551)
0812 14:52:36.203895 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:36.203897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:36.203990 (+ 93us) server_negotiation.cc:407] Connection header received
0812 14:52:36.204049 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:36.204049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:36.204063 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:36.204096 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:36.204397 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.204885 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.205544 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.205727 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.205779 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:36.208406 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:52019: BlockingRecv error: recv got EOF from 127.4.27.3:52019 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 14:52:36 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:36 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:36.979231 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:36.975007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:34637)
0812 14:52:36.975164 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:36.975175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:36.975237 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:36.975359 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:36.975359 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:36.975508 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:36.975559 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:36.975560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.976144 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.976144 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:36.976667 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:36.976668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.976944 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.976944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:36.976980 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:36.977485 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:36.977490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:36.979158 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
W20260812 14:52:36.979321 20808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:36.975082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:34637 (local address 127.4.27.62:36551)
0812 14:52:36.975190 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:36.975192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:36.975221 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:52:36.975284 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:36.975284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:36.975294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:36.975348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:36.975606 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.976094 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.976716 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:36.976900 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:36.976940 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:36.979258 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:34637: BlockingRecv error: recv got EOF from 127.4.27.1:34637 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
Aug 12 14:52:37 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:37 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:37.085175 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:37.081171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:49945)
0812 14:52:37.081338 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:37.081349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:37.081407 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:37.081521 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:37.081522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:37.081669 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:37.081717 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:37.081718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.082311 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.082311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:37.082837 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:37.082838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.083125 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.083126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:37.083160 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:37.083540 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:37.083545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:37.085121 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
W20260812 14:52:37.085306 20808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:37.081229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:49945 (local address 127.4.27.62:36551)
0812 14:52:37.081276 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:37.081279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:37.081404 (+ 125us) server_negotiation.cc:407] Connection header received
0812 14:52:37.081462 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:37.081462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:37.081474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:37.081511 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:37.081763 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.082254 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.082892 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.083072 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.083123 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:37.085236 (+ 2113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:49945: BlockingRecv error: recv got EOF from 127.4.27.2:49945 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 12 14:52:37 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:37 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:37.213163 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:37.208799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:58103)
0812 14:52:37.208950 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:37.208976 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:37.209038 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:37.209182 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:37.209183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:37.209355 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:37.209402 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:37.209403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.210023 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.210025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:37.210569 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:37.210571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.210888 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.210889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:37.210945 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:37.211406 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:37.211411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:37.213071 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
W20260812 14:52:37.213310 20808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:37.208850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:58103 (local address 127.4.27.62:36551)
0812 14:52:37.208914 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:37.208916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:37.209031 (+ 115us) server_negotiation.cc:407] Connection header received
0812 14:52:37.209089 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:37.209090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:37.209104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:37.209169 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:37.209472 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.209954 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.210644 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.210829 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.210884 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:37.213192 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:58103: BlockingRecv error: recv got EOF from 127.4.27.3:58103 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 14:52:37 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:37 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:37.988328 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:37.979666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:48001)
0812 14:52:37.979843 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:37.979853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:37.979910 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:37.980031 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:37.980031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:37.980188 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:37.980238 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:37.980240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.980830 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.980831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:37.981351 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:37.981352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.981653 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.981653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:37.981694 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:37.986585 (+ 4891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:37.986590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:37.988253 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
W20260812 14:52:37.988415 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:37.979728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:48001 (local address 127.4.27.62:36551)
0812 14:52:37.979848 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:37.979849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:37.979896 (+ 47us) server_negotiation.cc:407] Connection header received
0812 14:52:37.979961 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:37.979962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:37.979972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:37.980017 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:37.980288 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.980776 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.981413 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:37.981598 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:37.981645 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:37.988342 (+ 6697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:48001: BlockingRecv error: recv got EOF from 127.4.27.1:48001 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
Aug 12 14:52:38 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:38 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:38.089701 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:38.085550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:51753)
0812 14:52:38.085693 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:38.085706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:38.085774 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:38.085903 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:38.085904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:38.086064 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:38.086109 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:38.086111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.086786 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.086787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:38.087301 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:38.087303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.087624 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.087625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:38.087690 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:38.088068 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:38.088072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:38.089649 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":52,"threads_started":1}
W20260812 14:52:38.089854 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:38.085667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:51753 (local address 127.4.27.62:36551)
0812 14:52:38.085723 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:38.085726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:38.085754 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:52:38.085830 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:38.085831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:38.085843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:38.085893 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:38.086181 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.086709 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.087372 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.087562 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.087620 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:38.089796 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:51753: BlockingRecv error: recv got EOF from 127.4.27.2:51753 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 12 14:52:38 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:38 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:38.218230 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:38.213573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:43975)
0812 14:52:38.213689 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:38.213700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:38.213752 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:38.213892 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:38.213893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:38.214039 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:38.214087 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:38.214089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.215037 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.215037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:38.215554 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:38.215556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.216006 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.216007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:38.216040 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:38.216462 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:38.216466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:38.218149 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
W20260812 14:52:38.218279 20816 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:38.213643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:43975 (local address 127.4.27.62:36551)
0812 14:52:38.213699 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:38.213701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:38.213744 (+ 43us) server_negotiation.cc:407] Connection header received
0812 14:52:38.213815 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:38.213815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:38.213830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:38.213876 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:38.214148 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.214966 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.215628 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.215942 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.216005 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:38.218232 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:43975: BlockingRecv error: recv got EOF from 127.4.27.3:43975 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 14:52:38 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:38 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:38.993026 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:38.988732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:53187)
0812 14:52:38.988874 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:38.988884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:38.988938 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:38.989066 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:38.989066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:38.989224 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:38.989276 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:38.989277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.990030 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.990031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:38.990616 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:38.990619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.990911 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.990912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:38.990948 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:38.991340 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:38.991344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:38.992968 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20260812 14:52:38.993126 20822 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:38.988791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:53187 (local address 127.4.27.62:36551)
0812 14:52:38.988899 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:38.988901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:38.988923 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:52:38.988984 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:38.988985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:38.988996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:38.989053 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:38.989352 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.989956 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.990678 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:38.990863 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:38.990908 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:38.993060 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:53187: BlockingRecv error: recv got EOF from 127.4.27.1:53187 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
Aug 12 14:52:39 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:39 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:39.094501 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:39.090122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:42193)
0812 14:52:39.090249 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:39.090260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:39.090324 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:39.090484 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:39.090485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:39.090665 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:39.090710 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:39.090712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.091350 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.091351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:39.091915 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:39.091918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.092250 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.092251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:39.092306 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:39.092753 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:39.092759 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:39.094429 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
W20260812 14:52:39.094575 20822 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:39.090197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:42193 (local address 127.4.27.62:36551)
0812 14:52:39.090248 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:39.090250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:39.090305 (+ 55us) server_negotiation.cc:407] Connection header received
0812 14:52:39.090411 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:39.090412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:39.090425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:39.090469 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:39.090779 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.091277 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.091986 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.092186 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.092251 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:39.094519 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:42193: BlockingRecv error: recv got EOF from 127.4.27.2:42193 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 12 14:52:39 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:39 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:39.222807 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:39.218673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:54415)
0812 14:52:39.218808 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:39.218818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:39.218872 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:39.218989 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:39.218989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:39.219169 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:39.219219 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:39.219221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.219819 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.219820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:39.220363 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:39.220365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.220640 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.220641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:39.220676 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:39.221053 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:39.221057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:39.222733 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
W20260812 14:52:39.222882 20822 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:39.218741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:54415 (local address 127.4.27.62:36551)
0812 14:52:39.218791 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:39.218793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:39.218866 (+ 73us) server_negotiation.cc:407] Connection header received
0812 14:52:39.218925 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:39.218926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:39.218938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:39.218975 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:39.219269 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.219767 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.220420 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.220588 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.220639 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:39.222828 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:54415: BlockingRecv error: recv got EOF from 127.4.27.3:54415 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 12 14:52:39 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:39 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:39.997988 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:39.993479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:38319)
0812 14:52:39.993619 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:39.993642 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:39.993693 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:39.993858 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:39.993860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:39.994027 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:39.994080 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:39.994081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.994838 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.994839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:39.995378 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:39.995380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.995719 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.995719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:39.995756 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:39.996239 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:39.996243 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:39.997916 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
W20260812 14:52:39.998076 20833 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:39.993585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:38319 (local address 127.4.27.62:36551)
0812 14:52:39.993709 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:39.993711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:39.993718 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:39.993741 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:39.993741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:39.993754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:39.993816 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:39.994213 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.994742 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.995461 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:39.995662 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:39.995716 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:39.998016 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:38319: BlockingRecv error: recv got EOF from 127.4.27.1:38319 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
Aug 12 14:52:40 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:40 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:40.099376 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:40.094946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:55285)
0812 14:52:40.095098 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:40.095108 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:40.095160 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:40.095275 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:40.095276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:40.095433 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:40.095482 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:40.095484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.096132 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.096133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:40.096642 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:40.096643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.096911 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.096911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:40.096949 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:40.097354 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:40.097358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:40.099300 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
W20260812 14:52:40.099438 20833 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:40.095039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:55285 (local address 127.4.27.62:36551)
0812 14:52:40.095100 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:40.095102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:40.095147 (+ 45us) server_negotiation.cc:407] Connection header received
0812 14:52:40.095205 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:40.095206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:40.095216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:40.095264 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:40.095559 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.096056 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.096694 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.096865 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.096909 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:40.099386 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:55285: BlockingRecv error: recv got EOF from 127.4.27.2:55285 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 12 14:52:40 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:40 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:40.227478 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:40.223296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:44701)
0812 14:52:40.223439 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:40.223449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:40.223511 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:40.223633 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:40.223634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:40.223802 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:40.223851 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:40.223853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.224475 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.224476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:40.224992 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:40.224994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.225269 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.225269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:40.225303 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:40.225705 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:40.225710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:40.227419 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260812 14:52:40.227695 20833 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:40.223398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:44701 (local address 127.4.27.62:36551)
0812 14:52:40.223448 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:40.223450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:40.223498 (+ 48us) server_negotiation.cc:407] Connection header received
0812 14:52:40.223554 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:40.223555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:40.223568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:40.223620 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:40.223924 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.224415 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.225052 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.225225 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.225267 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:40.227627 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:44701: BlockingRecv error: recv got EOF from 127.4.27.3:44701 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 14:52:40 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:41 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:41.002938 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:40.998500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:45687)
0812 14:52:40.998646 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:40.998658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:40.998717 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:40.998855 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:40.998856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:40.999041 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:40.999092 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:40.999094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.999751 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:40.999752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:41.000289 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:41.000291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.000616 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.000617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:41.000667 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:41.001094 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:41.001098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:41.002848 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260812 14:52:41.003054 20843 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:40.998612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:45687 (local address 127.4.27.62:36551)
0812 14:52:40.998739 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:40.998741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:40.998750 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:40.998774 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:40.998774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:40.998787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:40.998838 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:40.999168 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:40.999675 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.000367 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.000556 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.000617 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:41.002945 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:45687: BlockingRecv error: recv got EOF from 127.4.27.1:45687 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Aug 12 14:52:41 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:41 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:41.104290 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:41.099826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:58987)
0812 14:52:41.099951 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:41.099961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:41.100030 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:41.100204 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:41.100205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:41.100368 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:41.100419 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:41.100421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.101233 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.101234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:41.101773 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:41.101775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.102073 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.102074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:41.102116 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:41.102575 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:41.102579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:41.104232 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
W20260812 14:52:41.104435 20843 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:41.099915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:58987 (local address 127.4.27.62:36551)
0812 14:52:41.099989 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:41.099991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:41.100019 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:52:41.100092 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:41.100093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:41.100109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:41.100163 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:41.100484 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.101143 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.101836 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.102030 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.102070 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:41.104366 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:58987: BlockingRecv error: recv got EOF from 127.4.27.2:58987 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 14:52:41 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:41 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:41.232191 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:41.227935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:42967)
0812 14:52:41.228073 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:41.228086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:41.228151 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:41.228288 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:41.228289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:41.228464 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:41.228510 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:41.228511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.229138 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.229140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:41.229654 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:41.229655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.229976 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.229978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:41.230031 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:41.230507 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:41.230511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:41.232117 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260812 14:52:41.232278 20843 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:41.228013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:42967 (local address 127.4.27.62:36551)
0812 14:52:41.228077 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:41.228079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:41.228132 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:52:41.228202 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:41.228203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:41.228217 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:41.228277 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:41.228579 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.229068 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.229725 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:41.229916 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:41.229970 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:41.232214 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:42967: BlockingRecv error: recv got EOF from 127.4.27.3:42967 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 12 14:52:42 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:42 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:42.008319 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:42.003385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:37825)
0812 14:52:42.003507 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:42.003521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:42.003585 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:42.003897 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:42.003899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:42.004137 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:42.004218 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:42.004220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.004968 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.004970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:42.005541 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:42.005543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.005889 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.005889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:42.005926 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:42.006406 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:42.006410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:42.008243 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
W20260812 14:52:42.008457 20850 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:42.003452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:37825 (local address 127.4.27.62:36551)
0812 14:52:42.003652 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:42.003654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:42.003662 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:52:42.003684 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:42.003685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:42.003696 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:42.003763 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:42.004342 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.004858 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.005635 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.005833 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.005885 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:42.008345 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:37825: BlockingRecv error: recv got EOF from 127.4.27.1:37825 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
Aug 12 14:52:42 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:42 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:42.109241 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:42.104750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:43675)
0812 14:52:42.104878 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:42.104890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:42.104964 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:42.105097 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:42.105098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:42.105283 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:42.105330 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:42.105333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.105970 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.105971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:42.106534 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:42.106537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.106864 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.106866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:42.106918 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:42.107322 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:42.107326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:42.109149 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
W20260812 14:52:42.109302 20850 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:42.104836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:43675 (local address 127.4.27.62:36551)
0812 14:52:42.104896 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:42.104898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:42.104950 (+ 52us) server_negotiation.cc:407] Connection header received
0812 14:52:42.105022 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:42.105023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:42.105037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:42.105079 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:42.105400 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.105903 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.106605 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.106805 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.106860 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:42.109249 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:43675: BlockingRecv error: recv got EOF from 127.4.27.2:43675 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 12 14:52:42 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:42 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:42.236994 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:42.232672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:32841)
0812 14:52:42.232828 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:42.232840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:42.232909 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:42.233022 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:42.233023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:42.233180 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:42.233231 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:42.233232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.233827 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.233828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:42.234404 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:42.234406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.234680 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.234681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:42.234716 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:42.235097 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:42.235100 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:42.236920 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
W20260812 14:52:42.237115 20850 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:42.232781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:32841 (local address 127.4.27.62:36551)
0812 14:52:42.232830 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:42.232832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:42.232891 (+ 59us) server_negotiation.cc:407] Connection header received
0812 14:52:42.232950 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:42.232950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:42.232962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:42.233005 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:42.233291 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.233775 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.234460 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:42.234630 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:42.234671 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:42.237014 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:32841: BlockingRecv error: recv got EOF from 127.4.27.3:32841 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 12 14:52:43 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:43 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:43.017874 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:43.008773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:33911)
0812 14:52:43.008896 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:43.008906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:43.008957 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:43.009092 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:43.009092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:43.009265 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:43.009321 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:43.009323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.009937 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.009939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:43.010617 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:43.010619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.010979 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.010979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:43.011017 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:43.016014 (+ 4997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:43.016018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:43.017794 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
W20260812 14:52:43.017963 20862 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:43.008830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:33911 (local address 127.4.27.62:36551)
0812 14:52:43.008973 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:43.008975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:43.008985 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:43.009006 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:43.009007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:43.009018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:43.009075 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:43.009376 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.009863 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.010710 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.010913 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.010970 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:43.017900 (+ 6930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:33911: BlockingRecv error: recv got EOF from 127.4.27.1:33911 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
Aug 12 14:52:43 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:43 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:43.114110 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:43.109676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:47281)
0812 14:52:43.109847 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:43.109859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:43.109931 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:43.110075 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:43.110076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:43.110250 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:43.110298 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:43.110300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.110977 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.110978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:43.111509 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:43.111511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.111839 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.111840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:43.111894 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:43.112394 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:43.112398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:43.114036 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
W20260812 14:52:43.114198 20862 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:43.109760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:47281 (local address 127.4.27.62:36551)
0812 14:52:43.109856 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:43.109857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:43.109918 (+ 61us) server_negotiation.cc:407] Connection header received
0812 14:52:43.110001 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:43.110002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:43.110017 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:43.110059 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:43.110402 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.110905 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.111585 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.111777 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.111839 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:43.114137 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:47281: BlockingRecv error: recv got EOF from 127.4.27.2:47281 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 12 14:52:43 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:43 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:43.241725 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:43.237440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:51025)
0812 14:52:43.237593 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:43.237607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:43.237662 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:43.237775 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:43.237775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:43.237935 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:43.237981 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:43.237982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.238598 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.238598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:43.239131 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:43.239133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.239430 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.239432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:43.239486 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:43.239958 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:43.239963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:43.241647 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
W20260812 14:52:43.241787 20862 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:43.237575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:51025 (local address 127.4.27.62:36551)
0812 14:52:43.237637 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:43.237639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:43.237649 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:43.237706 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:43.237707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:43.237719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:43.237764 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:43.238030 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.238539 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.239189 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:43.239356 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:43.239398 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:43.241734 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:51025: BlockingRecv error: recv got EOF from 127.4.27.3:51025 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 12 14:52:44 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:44 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:44.022552 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:44.018377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:43085)
0812 14:52:44.018505 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:44.018517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:44.018576 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:44.018699 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:44.018699 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:44.018860 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:44.018911 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:44.018912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.019501 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.019501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:44.020031 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:44.020033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.020305 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.020305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:44.020341 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:44.020732 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:44.020735 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:44.022491 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
W20260812 14:52:44.022639 20869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:44.018424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:43085 (local address 127.4.27.62:36551)
0812 14:52:44.018562 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:44.018563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:44.018572 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:44.018630 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:44.018631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:44.018641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:44.018685 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:44.018964 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.019449 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.020083 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.020259 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.020300 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:44.022578 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:43085: BlockingRecv error: recv got EOF from 127.4.27.1:43085 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
Aug 12 14:52:44 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:44 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:44.118676 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:44.114530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:54867)
0812 14:52:44.114657 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:44.114668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:44.114736 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:44.114887 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:44.114888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:44.115047 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:44.115091 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:44.115093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.115721 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.115722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:44.116239 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:44.116241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.116569 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.116570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:44.116629 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:44.117010 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:44.117015 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:44.118590 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
W20260812 14:52:44.118753 20869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:44.114599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:54867 (local address 127.4.27.62:36551)
0812 14:52:44.114655 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:44.114657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:44.114723 (+ 66us) server_negotiation.cc:407] Connection header received
0812 14:52:44.114796 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:44.114797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:44.114810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:44.114869 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:44.115159 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.115649 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.116311 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.116509 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.116564 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:44.118696 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:54867: BlockingRecv error: recv got EOF from 127.4.27.2:54867 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 14:52:44 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:44 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:44.246186 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:44.242176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:36777)
0812 14:52:44.242290 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:44.242301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:44.242390 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:44.242517 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:44.242517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:44.242676 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:44.242727 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:44.242728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.243309 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.243309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:44.243842 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:44.243843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.244109 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.244110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:44.244146 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:44.244576 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:44.244580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:44.246128 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260812 14:52:44.246310 20869 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:44.242252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:36777 (local address 127.4.27.62:36551)
0812 14:52:44.242326 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:44.242328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:44.242363 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:52:44.242439 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:44.242439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:44.242453 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:44.242505 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:44.242772 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.243259 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.243891 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:44.244064 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:44.244104 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:44.246252 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:36777: BlockingRecv error: recv got EOF from 127.4.27.3:36777 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 14:52:45 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:45 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:45.027441 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:45.023041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:38659)
0812 14:52:45.023177 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:45.023187 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:45.023238 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:45.023390 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:45.023391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:45.023571 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:45.023625 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:45.023627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.024294 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.024295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:45.024811 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:45.024813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.025148 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.025149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:45.025222 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:45.025630 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:45.025634 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:45.027363 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260812 14:52:45.027539 20885 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:45.023111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:38659 (local address 127.4.27.62:36551)
0812 14:52:45.023253 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:45.023255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:45.023264 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:45.023288 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:45.023288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:45.023300 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:45.023362 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:45.023709 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.024217 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.024889 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.025085 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.025140 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:45.027476 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:38659: BlockingRecv error: recv got EOF from 127.4.27.1:38659 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
Aug 12 14:52:45 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:45 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:45.123389 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:45.119069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:54333)
0812 14:52:45.119199 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:45.119210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:45.119287 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:45.119404 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:45.119405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:45.119571 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:45.119617 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:45.119618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.120254 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.120255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:45.120773 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:45.120775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.121116 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.121117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:45.121176 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:45.121629 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:45.121634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:45.123304 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260812 14:52:45.123457 20885 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:45.119143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:54333 (local address 127.4.27.62:36551)
0812 14:52:45.119195 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:45.119197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:45.119267 (+ 70us) server_negotiation.cc:407] Connection header received
0812 14:52:45.119339 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:45.119340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:45.119354 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:45.119399 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:45.119687 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.120179 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.120848 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.121046 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.121111 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:45.123403 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:54333: BlockingRecv error: recv got EOF from 127.4.27.2:54333 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 12 14:52:45 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:45 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:45.250738 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:45.246674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:51195)
0812 14:52:45.246802 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:45.246813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:45.246875 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:45.246986 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:45.246987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:45.247142 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:45.247207 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:45.247208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.247809 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.247809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:45.248322 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:45.248324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.248657 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.248657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:45.248694 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:45.249078 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:45.249081 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:45.250660 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260812 14:52:45.250811 20885 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:45.246730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:51195 (local address 127.4.27.62:36551)
0812 14:52:45.246791 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:45.246793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:45.246870 (+ 77us) server_negotiation.cc:407] Connection header received
0812 14:52:45.246927 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:45.246927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:45.246939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:45.246973 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:45.247256 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.247736 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.248409 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:45.248599 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:45.248656 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:45.250756 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:51195: BlockingRecv error: recv got EOF from 127.4.27.3:51195 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 12 14:52:46 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:46 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:46.032262 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:46.027939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:40419)
0812 14:52:46.028066 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:46.028076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:46.028148 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:46.028363 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:46.028364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:46.028524 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:46.028575 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:46.028577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.029278 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.029279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:46.029816 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:46.029817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.030100 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.030101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:46.030141 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:46.030552 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:46.030556 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:46.032192 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
W20260812 14:52:46.032362 20890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:46.028037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:40419 (local address 127.4.27.62:36551)
0812 14:52:46.028223 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:46.028225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:46.028235 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:52:46.028259 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:46.028260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:46.028272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:46.028349 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:46.028664 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.029181 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.029873 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.030055 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.030098 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:46.032289 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:40419: BlockingRecv error: recv got EOF from 127.4.27.1:40419 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
Aug 12 14:52:46 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:46 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:46.128103 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:46.123789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:46409)
0812 14:52:46.123922 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:46.123934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:46.124000 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:46.124126 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:46.124127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:46.124296 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:46.124341 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:46.124343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.124972 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.124974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:46.125520 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:46.125522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.125852 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.125853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:46.125911 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:46.126327 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:46.126332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:46.128035 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
W20260812 14:52:46.128177 20890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:46.123880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:46409 (local address 127.4.27.62:36551)
0812 14:52:46.123938 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:46.123940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:46.123979 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:52:46.124057 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:46.124058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:46.124070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:46.124112 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:46.124408 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.124900 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.125595 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.125790 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.125845 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:46.128119 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:46409: BlockingRecv error: recv got EOF from 127.4.27.2:46409 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:52:46 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:46 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:46.255523 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:46.251161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:58937)
0812 14:52:46.251327 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:46.251338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:46.251403 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:46.251534 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:46.251535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:46.251700 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:46.251747 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:46.251749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.252379 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.252380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:46.252890 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:46.252892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.253210 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.253211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:46.253265 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:46.253665 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:46.253669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:46.255436 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
W20260812 14:52:46.255667 20890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:46.251244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:58937 (local address 127.4.27.62:36551)
0812 14:52:46.251310 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:46.251312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:46.251394 (+ 82us) server_negotiation.cc:407] Connection header received
0812 14:52:46.251465 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:46.251466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:46.251478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:46.251518 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:46.251818 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.252309 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.252966 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:46.253150 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:46.253203 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:46.255594 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:58937: BlockingRecv error: recv got EOF from 127.4.27.3:58937 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 12 14:52:47 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:47 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:47.036796 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:47.032700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:33895)
0812 14:52:47.032826 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:47.032838 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:47.032890 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:47.033009 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:47.033010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:47.033173 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:47.033224 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:47.033225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.033814 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.033815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:47.034333 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:47.034334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.034629 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.034630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:47.034670 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:47.035077 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:47.035081 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:47.036739 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20260812 14:52:47.036900 20896 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:47.032767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:33895 (local address 127.4.27.62:36551)
0812 14:52:47.032899 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:47.032900 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:47.032909 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:47.032930 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:47.032930 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:47.032941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:47.032998 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:47.033275 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.033762 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.034403 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.034581 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.034627 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:47.036834 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:33895: BlockingRecv error: recv got EOF from 127.4.27.1:33895 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
Aug 12 14:52:47 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:47 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:47.132643 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:47.128503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:37697)
0812 14:52:47.128636 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:47.128647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:47.128707 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:47.128817 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:47.128817 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:47.128963 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:47.129010 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:47.129011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.129655 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.129656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:47.130182 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:47.130184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.130471 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.130472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:47.130509 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:47.130905 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:47.130909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:47.132569 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260812 14:52:47.132732 20896 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:47.128559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:37697 (local address 127.4.27.62:36551)
0812 14:52:47.128632 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:47.128634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:47.128704 (+ 70us) server_negotiation.cc:407] Connection header received
0812 14:52:47.128759 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:47.128759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:47.128770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:47.128807 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:47.129098 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.129596 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.130237 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.130428 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.130472 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:47.132674 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:37697: BlockingRecv error: recv got EOF from 127.4.27.2:37697 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 12 14:52:47 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:47 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.3@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:47.260411 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:47.255972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.3:54547)
0812 14:52:47.256126 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:47.256137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:47.256208 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:47.256334 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:47.256335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:47.256512 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:47.256559 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:47.256561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.257181 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.257182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:47.257716 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:47.257718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.258035 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.258037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:47.258089 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:47.258558 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:47.258563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:47.260335 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
W20260812 14:52:47.260474 20896 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:47.256063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:54547 (local address 127.4.27.62:36551)
0812 14:52:47.256125 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:47.256127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:47.256187 (+ 60us) server_negotiation.cc:407] Connection header received
0812 14:52:47.256260 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:47.256261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:47.256275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:47.256321 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:47.256631 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.257111 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.257794 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:47.257975 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:47.258030 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:47.260422 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.3:54547: BlockingRecv error: recv got EOF from 127.4.27.3:54547 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 14:52:48 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:48 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.1@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:48.045221 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.037386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.1:43091)
0812 14:52:48.037529 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:48.037541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:48.037608 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:48.037761 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:48.037762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:48.037933 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:48.037981 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.037983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.038699 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.038700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.039215 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.039217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.039547 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.039548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.039602 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.043497 (+ 3895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:48.043502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:48.045165 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
W20260812 14:52:48.045318 20902 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:48.037455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:43091 (local address 127.4.27.62:36551)
0812 14:52:48.037563 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.037564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.037593 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:52:48.037675 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.037676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.037689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.037748 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.038050 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.038623 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.039285 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.039485 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.039541 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.045254 (+ 5713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.1:43091: BlockingRecv error: recv got EOF from 127.4.27.1:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
Aug 12 14:52:48 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 14:52:48 dist-test-slave-6zbq krb5kdc[18938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.27.2@KRBTEST.COM for oryx/127.4.27.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 14:52:48.137535 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.133174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36551 (local address 127.4.27.2:38853)
0812 14:52:48.133315 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:48.133328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:48.133399 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:48.133535 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:48.133536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:48.133710 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:48.133759 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.133761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.134421 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.134423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.134980 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.134982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.135319 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.135320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.135376 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.135797 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:48.135802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:48.137460 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.27.62:36551: Server oryx/127.4.27.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
W20260812 14:52:48.137612 20902 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:48.133272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:38853 (local address 127.4.27.62:36551)
0812 14:52:48.133330 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.133332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.133379 (+ 47us) server_negotiation.cc:407] Connection header received
0812 14:52:48.133460 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.133461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.133472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.133514 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.133831 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.134320 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.135061 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.135259 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.135313 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.137559 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.27.2:38853: BlockingRecv error: recv got EOF from 127.4.27.2:38853 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:52:48.195515 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 20029
I20260812 14:52:48.206897 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 20289
I20260812 14:52:48.218333 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 20542
I20260812 14:52:48.227362 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 19893
2026-08-12T14:52:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17565 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T14:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:48Z Disabled control of system clock
I20260812 14:52:48.257711 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:45317
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:45317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 14:52:48.335413 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:48.335594 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:48.335623 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:48.336788 20918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:48.336823 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:48.336834 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:48.336845 20918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:48.336856 20918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:48.338131 20918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:45317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:45317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:48.338307 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:48.338587 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:48.343235 20930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.343304 20918 server_base.cc:1034] running on GCE node
W20260812 14:52:48.343294 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:48.343282 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.343663 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:48.343928 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:48.345067 20918 hybrid_clock.cc:648] HybridClock initialized: now 1786546368345052 us; error 28 us; skew 500 ppm
I20260812 14:52:48.346167 20918 webserver.cc:460] Webserver started at http://127.4.27.62:46781/ using document root <none> and password file <none>
I20260812 14:52:48.346411 20918 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:48.346462 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:48.346586 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:48.347553 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "1eac6e31fe6547bdad87d5c7d6f61263"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "4d0cc8c0a7259bfe168dbad613c91427"
server_key_iv: "937b2407fada3d5beaea583c47bda9bc"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.347873 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "1eac6e31fe6547bdad87d5c7d6f61263"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "4d0cc8c0a7259bfe168dbad613c91427"
server_key_iv: "937b2407fada3d5beaea583c47bda9bc"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.349013 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:48.349625 20941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.349772 20918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:48.349854 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "1eac6e31fe6547bdad87d5c7d6f61263"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "4d0cc8c0a7259bfe168dbad613c91427"
server_key_iv: "937b2407fada3d5beaea583c47bda9bc"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.349931 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:48.361272 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:48.361493 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:48.361588 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:48.365064 20918 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:45317
I20260812 14:52:48.365095 21037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:45317 every 8 connection(s)
I20260812 14:52:48.365377 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:48.365985 21038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:48.368383 21038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263: Bootstrap starting.
I20260812 14:52:48.368937 21038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:48.369247 21038 log.cc:824] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:48.369848 21038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263: No bootstrap required, opened a new log
I20260812 14:52:48.371459 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 20918
I20260812 14:52:48.371549 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:48.371433 21038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45317 } }
I20260812 14:52:48.371589 21038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:48.371634 21038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eac6e31fe6547bdad87d5c7d6f61263, State: Initialized, Role: FOLLOWER
I20260812 14:52:48.371760 4204 external_mini_cluster.cc:1428] Setting key 6726e2ea8d0fb1d43ca790fc39e33e0d
I20260812 14:52:48.371795 21038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45317 } }
I20260812 14:52:48.371878 21038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:48.371913 21038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:48.371959 21038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:48.372579 21038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45317 } }
I20260812 14:52:48.372697 21038 leader_election.cc:302] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eac6e31fe6547bdad87d5c7d6f61263; no voters:
I20260812 14:52:48.372864 21038 leader_election.cc:288] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:48.372900 21046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:48.373052 21046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [term 1 LEADER]: Becoming Leader. State: Replica: 1eac6e31fe6547bdad87d5c7d6f61263, State: Running, Role: LEADER
I20260812 14:52:48.373122 21038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:48.373162 21046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45317 } }
I20260812 14:52:48.373569 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45317 } } }
I20260812 14:52:48.373551 21051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eac6e31fe6547bdad87d5c7d6f61263. Latest consensus state: current_term: 1 leader_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eac6e31fe6547bdad87d5c7d6f61263" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 45317 } } }
I20260812 14:52:48.373692 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:48.373698 21051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:48.373881 21058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:48.374517 21058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:48.375970 21058 catalog_manager.cc:1347] Generated new cluster ID: 7c8dcfdbd4ff4afa83ea6e89dd3dccaf
I20260812 14:52:48.376024 21058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:48.378322 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.372163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.4.27.62:45317)
0812 14:52:48.372382 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.372390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.372423 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:52:48.372536 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.372543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.372620 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.372760 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.373047 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.375391 (+ 2344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.376454 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.377020 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.377120 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.377679 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:48.377683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:48.377683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:48.377867 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:48.377905 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:48.377917 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:48.377975 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:48.378058 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:52:48.378127 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260812 14:52:48.389070 21058 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:48.389734 21058 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:48.399552 21058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eac6e31fe6547bdad87d5c7d6f61263: Generated new TSK 0
I20260812 14:52:48.399724 21058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:48.402474 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45317
--builtin_ntp_servers=127.4.27.20:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:52:48.488108 21080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:48.488315 21080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:48.488335 21080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:48.489459 21080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:48.489496 21080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:48.489519 21080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:48.490866 21080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:48.491086 21080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:48.491356 21080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:48.496156 21089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:48.496291 21092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:48.496156 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.496470 21080 server_base.cc:1034] running on GCE node
I20260812 14:52:48.496713 21080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:48.496977 21080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:48.498126 21080 hybrid_clock.cc:648] HybridClock initialized: now 1786546368498111 us; error 32 us; skew 500 ppm
I20260812 14:52:48.499328 21080 webserver.cc:460] Webserver started at http://127.4.27.1:33915/ using document root <none> and password file <none>
I20260812 14:52:48.499557 21080 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:48.499603 21080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:48.499718 21080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:48.500694 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "ccc115d7b3a2420eb81f8b61e72a7f2b"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "9660a091b0ba04b980f62c061c34c569"
server_key_iv: "dab25ce34b0a09d343ed88c715963b1a"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.501006 21080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "ccc115d7b3a2420eb81f8b61e72a7f2b"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "9660a091b0ba04b980f62c061c34c569"
server_key_iv: "dab25ce34b0a09d343ed88c715963b1a"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.502250 21080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:48.502996 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.503151 21080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.503252 21080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "ccc115d7b3a2420eb81f8b61e72a7f2b"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "9660a091b0ba04b980f62c061c34c569"
server_key_iv: "dab25ce34b0a09d343ed88c715963b1a"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.503319 21080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:48.513546 21080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:48.513796 21080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:48.513916 21080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:48.514102 21080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:48.514420 21080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:48.514462 21080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.514490 21080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:48.514513 21080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.519861 21080 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43331
I20260812 14:52:48.519930 21288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43331 every 8 connection(s)
I20260812 14:52:48.520190 21080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:48.524933 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.520819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:46867 (local address 127.4.27.62:45317)
0812 14:52:48.520873 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.520875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.521108 (+ 233us) server_negotiation.cc:407] Connection header received
0812 14:52:48.521630 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.521631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.521645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.521698 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.522300 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.522831 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.523753 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.523952 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.524008 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.524678 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:48.524680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:48.524680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:48.524684 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:48.524700 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:48.524704 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:48.524754 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:48.524825 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 14:52:48.524857 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:52:48.525100 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.520806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:45317 (local address 127.4.27.1:46867)
0812 14:52:48.520998 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:48.521028 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:48.521540 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:48.521760 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:48.521764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:48.521774 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:48.522211 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.522216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.522919 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.522920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.523665 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.523667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.524025 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.524026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.524151 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.524558 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:48.524559 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:48.524584 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:48.524604 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:48.524755 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:48.524761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:48.524764 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:48.524823 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:52:48.524920 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260812 14:52:48.525543 21290 heartbeater.cc:347] Connected to a master server at 127.4.27.62:45317
I20260812 14:52:48.525640 21290 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:48.525804 21290 heartbeater.cc:511] Master 127.4.27.62:45317 requested a full tablet report, sending...
I20260812 14:52:48.526263 20964 ts_manager.cc:194] Registered new tserver with Master: ccc115d7b3a2420eb81f8b61e72a7f2b (127.4.27.1:43331)
I20260812 14:52:48.526862 20964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.1:46867
I20260812 14:52:48.528445 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 21080
I20260812 14:52:48.528519 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:48.528664 4204 external_mini_cluster.cc:1428] Setting key bc4a8abb9a902e93aadc062c361eef43
I20260812 14:52:48.529409 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45317
--builtin_ntp_servers=127.4.27.20:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 14:52:48.609692 21295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:48.609936 21295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:48.609969 21295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:48.611562 21295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:48.611610 21295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:48.611644 21295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:48.613133 21295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:48.613351 21295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:48.613605 21295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:48.618152 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:48.618263 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.618170 21295 server_base.cc:1034] running on GCE node
W20260812 14:52:48.618204 21308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.618559 21295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:48.618767 21295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:48.619905 21295 hybrid_clock.cc:648] HybridClock initialized: now 1786546368619893 us; error 31 us; skew 500 ppm
I20260812 14:52:48.621053 21295 webserver.cc:460] Webserver started at http://127.4.27.2:35517/ using document root <none> and password file <none>
I20260812 14:52:48.621258 21295 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:48.621300 21295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:48.621414 21295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:48.622429 21295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "55c098dda57d4448a6310b9ec84a1e22"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "823819df564a1c85032a5d815ce02e6f"
server_key_iv: "39094163923230b1caf4982d470aa8d1"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.622750 21295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "55c098dda57d4448a6310b9ec84a1e22"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "823819df564a1c85032a5d815ce02e6f"
server_key_iv: "39094163923230b1caf4982d470aa8d1"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.623888 21295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:48.624497 21319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.624631 21295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:48.624716 21295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "55c098dda57d4448a6310b9ec84a1e22"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "823819df564a1c85032a5d815ce02e6f"
server_key_iv: "39094163923230b1caf4982d470aa8d1"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.624778 21295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:48.636029 21295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:48.636258 21295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:48.636374 21295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:48.636559 21295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:48.636845 21295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:48.636878 21295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.636898 21295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:48.636920 21295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.642287 21295 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:41787
I20260812 14:52:48.642330 21512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:41787 every 8 connection(s)
I20260812 14:52:48.642656 21295 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:48.644038 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 21295
I20260812 14:52:48.644136 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:48.644315 4204 external_mini_cluster.cc:1428] Setting key a81233f57c6036af290077ab76ca0445
I20260812 14:52:48.645184 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45317
--builtin_ntp_servers=127.4.27.20:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 14:52:48.647841 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.643275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:49349 (local address 127.4.27.62:45317)
0812 14:52:48.643355 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.643357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.643525 (+ 168us) server_negotiation.cc:407] Connection header received
0812 14:52:48.643963 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.643964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.643977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.644038 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.644759 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.645333 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.646605 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.646792 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.646850 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.647591 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:48.647592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:48.647593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:48.647597 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:48.647613 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:48.647616 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:48.647660 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:48.647739 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:52:48.647766 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:52:48.648030 21516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.643257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:45317 (local address 127.4.27.2:49349)
0812 14:52:48.643421 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:48.643452 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:48.643876 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:48.644093 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:48.644097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:48.644104 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:48.644675 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.644681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.645417 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.645418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.646533 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.646536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.646869 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.646870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.647012 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.647467 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:48.647467 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:48.647492 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:48.647512 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:48.647665 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:48.647670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:48.647672 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:48.647735 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:52:48.647840 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260812 14:52:48.648442 21513 heartbeater.cc:347] Connected to a master server at 127.4.27.62:45317
I20260812 14:52:48.648535 21513 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:48.648715 21513 heartbeater.cc:511] Master 127.4.27.62:45317 requested a full tablet report, sending...
I20260812 14:52:48.649045 20964 ts_manager.cc:194] Registered new tserver with Master: 55c098dda57d4448a6310b9ec84a1e22 (127.4.27.2:41787)
I20260812 14:52:48.649453 20964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.2:49349
W20260812 14:52:48.724762 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:48.724946 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:48.724965 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:48.726397 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:48.726434 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:48.726457 21517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:48.727871 21517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:45317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:48.728099 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:48.728369 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:48.733557 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.733662 21517 server_base.cc:1034] running on GCE node
W20260812 14:52:48.733566 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:48.733564 21526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.733989 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:48.734232 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:48.735375 21517 hybrid_clock.cc:648] HybridClock initialized: now 1786546368735356 us; error 38 us; skew 500 ppm
I20260812 14:52:48.736495 21517 webserver.cc:460] Webserver started at http://127.4.27.3:42833/ using document root <none> and password file <none>
I20260812 14:52:48.736711 21517 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:48.736758 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:48.736886 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:48.737897 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "de2ac2b52e774e1d8dfee4cdf4586666"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "114a060b0b7c74feebb6ec2bd44ddd65"
server_key_iv: "83c4f44a5d9b306ef7185c08f25de7ca"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.738231 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "de2ac2b52e774e1d8dfee4cdf4586666"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "114a060b0b7c74feebb6ec2bd44ddd65"
server_key_iv: "83c4f44a5d9b306ef7185c08f25de7ca"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.739423 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:48.740125 21539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.740274 21517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:48.740358 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "de2ac2b52e774e1d8dfee4cdf4586666"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "114a060b0b7c74feebb6ec2bd44ddd65"
server_key_iv: "83c4f44a5d9b306ef7185c08f25de7ca"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.740425 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:48.758188 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:48.758491 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:48.758616 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:48.758818 21517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:48.759125 21517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:48.759167 21517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.759196 21517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:48.759213 21517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.764293 21517 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:34087
I20260812 14:52:48.764353 21736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:34087 every 8 connection(s)
I20260812 14:52:48.764612 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:48.769119 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.765272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:54853 (local address 127.4.27.62:45317)
0812 14:52:48.765325 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.765327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.765546 (+ 219us) server_negotiation.cc:407] Connection header received
0812 14:52:48.765994 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.765995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.766005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.766055 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.766692 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.767179 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.768043 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.768225 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.768272 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.768909 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:48.768910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:48.768911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:48.768915 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:48.768929 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:48.768933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:48.768971 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:48.769030 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 14:52:48.769055 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:48.769277 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.765255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:45317 (local address 127.4.27.3:54853)
0812 14:52:48.765439 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:48.765469 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:48.765917 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:48.766105 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:48.766108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:48.766115 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:48.766629 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.766633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.767238 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.767239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.767982 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:48.767984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:48.768290 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:48.768291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:48.768389 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:48.768795 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:48.768796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:48.768821 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:48.768840 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:48.768977 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:48.768979 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:48.768981 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:48.769024 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:52:48.769094 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260812 14:52:48.769727 21737 heartbeater.cc:347] Connected to a master server at 127.4.27.62:45317
I20260812 14:52:48.769811 21737 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:48.769949 21737 heartbeater.cc:511] Master 127.4.27.62:45317 requested a full tablet report, sending...
I20260812 14:52:48.770277 20964 ts_manager.cc:194] Registered new tserver with Master: de2ac2b52e774e1d8dfee4cdf4586666 (127.4.27.3:34087)
I20260812 14:52:48.770694 20964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.3:54853
I20260812 14:52:48.770750 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 21517
I20260812 14:52:48.770830 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:48.770985 4204 external_mini_cluster.cc:1428] Setting key 3b602c2121565ed4c19cc601fe67f74f
I20260812 14:52:48.771801 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:52:48.773518 21045 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:48.773025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.4.27.62:45317)
0812 14:52:48.773081 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.773083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.773195 (+ 112us) server_negotiation.cc:407] Connection header received
0812 14:52:48.773284 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.773285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.773296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.773337 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.773466 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54908: BlockingRecv error: recv got EOF from 127.0.0.1:54908 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:52:48.773593 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 21080
I20260812 14:52:48.788300 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 21295
I20260812 14:52:48.807585 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 21517
I20260812 14:52:48.846760 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 20918
2026-08-12T14:52:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (619 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T14:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:48Z Disabled control of system clock
I20260812 14:52:48.877207 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41475
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:52:48.954511 21750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:48.954695 21750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:48.954725 21750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:48.955858 21750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:48.955899 21750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:48.955917 21750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:48.955933 21750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:48.955950 21750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:48.957260 21750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:41475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:48.957491 21750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:48.957736 21750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:48.963531 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.963640 21750 server_base.cc:1034] running on GCE node
W20260812 14:52:48.963689 21760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:48.963533 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:48.964057 21750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:48.964344 21750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:48.965499 21750 hybrid_clock.cc:648] HybridClock initialized: now 1786546368965481 us; error 36 us; skew 500 ppm
I20260812 14:52:48.966719 21750 webserver.cc:460] Webserver started at http://127.4.27.62:37155/ using document root <none> and password file <none>
I20260812 14:52:48.966951 21750 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:48.967000 21750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:48.967127 21750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:48.968151 21750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "4dfabe34085941daa2ac64dd36b07dc0"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "129f052dded0ac25479446892d42890c"
server_key_iv: "bd4612aeac0f0a78752b17ec853a1116"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.968480 21750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "4dfabe34085941daa2ac64dd36b07dc0"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "129f052dded0ac25479446892d42890c"
server_key_iv: "bd4612aeac0f0a78752b17ec853a1116"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.969787 21750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:48.970438 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.970592 21750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:48.970674 21750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "4dfabe34085941daa2ac64dd36b07dc0"
format_stamp: "Formatted at 2026-08-12 14:52:48 on dist-test-slave-6zbq"
server_key: "129f052dded0ac25479446892d42890c"
server_key_iv: "bd4612aeac0f0a78752b17ec853a1116"
server_key_version: "encryptionkey@0"
I20260812 14:52:48.970727 21750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:48.973819 21750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:48.974021 21750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:48.974108 21750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:48.977666 21750 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:41475
I20260812 14:52:48.977710 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:41475 every 8 connection(s)
I20260812 14:52:48.977985 21750 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:48.978837 21870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:48.980836 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 21750
I20260812 14:52:48.980931 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:48.981146 4204 external_mini_cluster.cc:1428] Setting key 38b52f07f4fa860f6dbe6ca30768a326
I20260812 14:52:48.981763 21870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0: Bootstrap starting.
I20260812 14:52:48.982304 21870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:48.982596 21870 log.cc:824] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:48.983443 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:48.981647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.4.27.62:41475)
0812 14:52:48.981846 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:48.981854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:48.981887 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:52:48.982309 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:48.982316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:48.982389 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:48.982531 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:48.983064 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:48.983068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:48.983068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:48.983072 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:48.983118 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:48.983128 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:48.983200 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:48.983250 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:52:48.983305 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260812 14:52:48.983503 21870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0: No bootstrap required, opened a new log
I20260812 14:52:48.984850 21870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfabe34085941daa2ac64dd36b07dc0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41475 } }
I20260812 14:52:48.984978 21870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:48.985013 21870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfabe34085941daa2ac64dd36b07dc0, State: Initialized, Role: FOLLOWER
I20260812 14:52:48.985128 21870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfabe34085941daa2ac64dd36b07dc0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41475 } }
I20260812 14:52:48.985209 21870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:48.985250 21870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:48.985298 21870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:48.986014 21870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfabe34085941daa2ac64dd36b07dc0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41475 } }
I20260812 14:52:48.986107 21870 leader_election.cc:302] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dfabe34085941daa2ac64dd36b07dc0; no voters:
I20260812 14:52:48.986253 21870 leader_election.cc:288] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:48.986291 21877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:48.986456 21877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 4dfabe34085941daa2ac64dd36b07dc0, State: Running, Role: LEADER
I20260812 14:52:48.986512 21870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:48.986558 21877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfabe34085941daa2ac64dd36b07dc0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41475 } }
I20260812 14:52:48.986939 21879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dfabe34085941daa2ac64dd36b07dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfabe34085941daa2ac64dd36b07dc0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41475 } } }
I20260812 14:52:48.986946 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dfabe34085941daa2ac64dd36b07dc0. Latest consensus state: current_term: 1 leader_uuid: "4dfabe34085941daa2ac64dd36b07dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfabe34085941daa2ac64dd36b07dc0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41475 } } }
I20260812 14:52:48.987185 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:48.987399 21879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:48.987485 21894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:48.987884 21894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:48.989297 21894 catalog_manager.cc:1347] Generated new cluster ID: dd103f35c6a74da9bd33522f96566126
I20260812 14:52:48.989351 21894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:52:49.035516 21894 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:49.035694 21894 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:49.042604 21894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dfabe34085941daa2ac64dd36b07dc0: Generated new TSK 0
I20260812 14:52:49.042804 21894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:52:49.057360 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41475
--builtin_ntp_servers=127.4.27.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:52:49.141855 21923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:49.142038 21923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:49.142058 21923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:49.143219 21923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:49.143258 21923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:49.143283 21923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:49.144603 21923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:49.144814 21923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:49.145076 21923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:49.149820 21935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:49.149799 21941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:49.149796 21936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:49.149940 21923 server_base.cc:1034] running on GCE node
I20260812 14:52:49.150234 21923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:49.150489 21923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:49.151623 21923 hybrid_clock.cc:648] HybridClock initialized: now 1786546369151610 us; error 30 us; skew 500 ppm
I20260812 14:52:49.152845 21923 webserver.cc:460] Webserver started at http://127.4.27.1:39167/ using document root <none> and password file <none>
I20260812 14:52:49.153100 21923 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:49.153162 21923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:49.153295 21923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:49.154307 21923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "6b99940ca42c42c88ba266ef2de92e3f"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "c4c1bd1c0df534c6d8ceb2e418209566"
server_key_iv: "65c912ab2e34efb24be2fdd58c12e5db"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.154683 21923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "6b99940ca42c42c88ba266ef2de92e3f"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "c4c1bd1c0df534c6d8ceb2e418209566"
server_key_iv: "65c912ab2e34efb24be2fdd58c12e5db"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.155839 21923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:49.156484 21948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.156620 21923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:49.156703 21923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "6b99940ca42c42c88ba266ef2de92e3f"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "c4c1bd1c0df534c6d8ceb2e418209566"
server_key_iv: "65c912ab2e34efb24be2fdd58c12e5db"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.156772 21923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:49.160060 21923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:49.160281 21923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:49.160394 21923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:49.160583 21923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:49.160861 21923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:49.160890 21923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.160919 21923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:49.160943 21923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.166229 21923 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43569
I20260812 14:52:49.166289 22153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43569 every 8 connection(s)
I20260812 14:52:49.166563 21923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:52:49.169023 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:49.167194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:48815 (local address 127.4.27.62:41475)
0812 14:52:49.167241 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:49.167243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:49.167438 (+ 195us) server_negotiation.cc:407] Connection header received
0812 14:52:49.167840 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:49.167841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:49.167845 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:49.167880 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:49.168868 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:49.168869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:49.168869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:49.168871 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:49.168880 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:49.168883 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:49.168916 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:49.168963 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 14:52:49.168973 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:52:49.169183 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:49.167168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41475 (local address 127.4.27.1:48815)
0812 14:52:49.167333 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:49.167369 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:49.167754 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:49.167937 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:49.167940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:49.167948 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:49.168757 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:49.168758 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:49.168783 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:49.168808 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:49.168916 (+ 108us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:49.168918 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:49.168920 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:49.168963 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:52:49.169005 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260812 14:52:49.169610 22154 heartbeater.cc:347] Connected to a master server at 127.4.27.62:41475
I20260812 14:52:49.169689 22154 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:49.169826 22154 heartbeater.cc:511] Master 127.4.27.62:41475 requested a full tablet report, sending...
I20260812 14:52:49.170260 21806 ts_manager.cc:194] Registered new tserver with Master: 6b99940ca42c42c88ba266ef2de92e3f (127.4.27.1:43569)
I20260812 14:52:49.173393 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 21923
I20260812 14:52:49.173461 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:49.173655 4204 external_mini_cluster.cc:1428] Setting key eeeb973627df1eecf2e498ce320abf4c
I20260812 14:52:49.174456 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41475
--builtin_ntp_servers=127.4.27.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 14:52:49.254066 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:49.254246 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:49.254266 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:49.255357 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:49.255390 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:49.255415 22160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:49.256657 22160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:49.256826 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:49.257040 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:49.261777 22169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:49.261763 22173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:49.261756 22168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:49.261832 22160 server_base.cc:1034] running on GCE node
I20260812 14:52:49.262115 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:49.262334 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:49.263489 22160 hybrid_clock.cc:648] HybridClock initialized: now 1786546369263476 us; error 29 us; skew 500 ppm
I20260812 14:52:49.264559 22160 webserver.cc:460] Webserver started at http://127.4.27.2:44421/ using document root <none> and password file <none>
I20260812 14:52:49.264768 22160 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:49.264815 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:49.264940 22160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:49.265889 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "2a74c8ff87a24054abca4a6b941e135e"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "3869dbc8103d3dde6a89f1adc246f550"
server_key_iv: "2afb6945af2aadfd580db40b3f060245"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.266201 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "2a74c8ff87a24054abca4a6b941e135e"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "3869dbc8103d3dde6a89f1adc246f550"
server_key_iv: "2afb6945af2aadfd580db40b3f060245"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.267442 22160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:52:49.268055 22183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.268200 22160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:49.268280 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "2a74c8ff87a24054abca4a6b941e135e"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "3869dbc8103d3dde6a89f1adc246f550"
server_key_iv: "2afb6945af2aadfd580db40b3f060245"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.268343 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:49.271476 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:49.271795 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:49.271932 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:49.272136 22160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:49.272424 22160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:49.272459 22160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.272483 22160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:49.272511 22160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.277865 22160 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:40649
I20260812 14:52:49.277920 22408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:40649 every 8 connection(s)
I20260812 14:52:49.278187 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:52:49.279095 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 22160
I20260812 14:52:49.279192 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:49.279362 4204 external_mini_cluster.cc:1428] Setting key 1243f1e23a1717f440a3db87e86cdf7a
I20260812 14:52:49.280180 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41475
--builtin_ntp_servers=127.4.27.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 14:52:49.280975 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:49.278823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:34287 (local address 127.4.27.62:41475)
0812 14:52:49.278863 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:49.278864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:49.279089 (+ 225us) server_negotiation.cc:407] Connection header received
0812 14:52:49.279590 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:49.279591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:49.279598 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:49.279653 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:49.280756 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:49.280757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:49.280757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:49.280762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:49.280778 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:49.280782 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:49.280829 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:49.280895 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 14:52:49.280908 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:52:49.281157 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:49.278803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41475 (local address 127.4.27.2:34287)
0812 14:52:49.278986 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:49.279015 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:49.279485 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:49.279729 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:49.279733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:49.279742 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:49.280624 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:49.280624 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:49.280651 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:49.280677 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:49.280832 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:49.280837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:49.280840 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:49.280905 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 14:52:49.280961 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260812 14:52:49.281911 22409 heartbeater.cc:347] Connected to a master server at 127.4.27.62:41475
I20260812 14:52:49.281996 22409 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:49.282126 22409 heartbeater.cc:511] Master 127.4.27.62:41475 requested a full tablet report, sending...
I20260812 14:52:49.282469 21806 ts_manager.cc:194] Registered new tserver with Master: 2a74c8ff87a24054abca4a6b941e135e (127.4.27.2:40649)
W20260812 14:52:49.357805 22417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:49.357977 22417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:49.357997 22417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:49.359166 22417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:49.359205 22417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:49.359231 22417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:49.360558 22417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:49.360766 22417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:49.361014 22417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:49.365520 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:49.365497 22427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:49.365504 22426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:49.365531 22417 server_base.cc:1034] running on GCE node
I20260812 14:52:49.365861 22417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:49.366084 22417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:49.367218 22417 hybrid_clock.cc:648] HybridClock initialized: now 1786546369367209 us; error 41 us; skew 500 ppm
I20260812 14:52:49.368326 22417 webserver.cc:460] Webserver started at http://127.4.27.3:42323/ using document root <none> and password file <none>
I20260812 14:52:49.368534 22417 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:49.368580 22417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:49.368717 22417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:49.369635 22417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "c959b3187b3448d991804eccd1b7ddbf"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "fad2f6be57d4782fc7646cd2b2ccc804"
server_key_iv: "1823b4e8dc933d5a5bea7c26a698e13a"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.369946 22417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "c959b3187b3448d991804eccd1b7ddbf"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "fad2f6be57d4782fc7646cd2b2ccc804"
server_key_iv: "1823b4e8dc933d5a5bea7c26a698e13a"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.371160 22417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:52:49.371827 22448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.371973 22417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:49.372052 22417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "c959b3187b3448d991804eccd1b7ddbf"
format_stamp: "Formatted at 2026-08-12 14:52:49 on dist-test-slave-6zbq"
server_key: "fad2f6be57d4782fc7646cd2b2ccc804"
server_key_iv: "1823b4e8dc933d5a5bea7c26a698e13a"
server_key_version: "encryptionkey@0"
I20260812 14:52:49.372116 22417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:49.375439 22417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:49.375662 22417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:49.375775 22417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:49.375952 22417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:49.376206 22417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:49.376237 22417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.376261 22417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:49.376286 22417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:49.381451 22417 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:33625
I20260812 14:52:49.381486 22651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:33625 every 8 connection(s)
I20260812 14:52:49.381752 22417 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:49.384130 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:49.382373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:44307 (local address 127.4.27.62:41475)
0812 14:52:49.382420 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:49.382422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:49.382615 (+ 193us) server_negotiation.cc:407] Connection header received
0812 14:52:49.382988 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:49.382989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:49.382993 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:49.383035 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:49.383964 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:49.383965 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:49.383965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:52:49.383968 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:49.383981 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:52:49.383984 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:49.384023 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:49.384067 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:52:49.384077 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:52:49.384279 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:49.382335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41475 (local address 127.4.27.3:44307)
0812 14:52:49.382512 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:49.382539 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:49.382917 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:49.383088 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:49.383090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:49.383097 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:49.383849 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:52:49.383849 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:49.383874 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:52:49.383899 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:49.384024 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:49.384026 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:49.384028 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:49.384072 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:52:49.384115 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260812 14:52:49.384675 22652 heartbeater.cc:347] Connected to a master server at 127.4.27.62:41475
I20260812 14:52:49.384753 22652 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:49.384855 22652 heartbeater.cc:511] Master 127.4.27.62:41475 requested a full tablet report, sending...
I20260812 14:52:49.385142 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 22417
I20260812 14:52:49.385188 21806 ts_manager.cc:194] Registered new tserver with Master: c959b3187b3448d991804eccd1b7ddbf (127.4.27.3:33625)
I20260812 14:52:49.385228 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:49.385401 4204 external_mini_cluster.cc:1428] Setting key d0f8dc947dfe5205ed4e46f898e6e22e
I20260812 14:52:49.386116 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 14:52:49.387799 21874 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:52:49.387366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.4.27.62:41475)
0812 14:52:49.387396 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:49.387398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:49.387525 (+ 127us) server_negotiation.cc:407] Connection header received
0812 14:52:49.387587 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:49.387588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:49.387592 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:49.387636 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:49.387748 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37504: BlockingRecv error: recv got EOF from 127.0.0.1:37504 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 14:52:49.387846 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 21923
I20260812 14:52:49.401268 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 22160
I20260812 14:52:49.426234 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 22417
I20260812 14:52:49.432945 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 21750
2026-08-12T14:52:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (585 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:49 dist-test-slave-6zbq krb5kdc[22671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:49 dist-test-slave-6zbq krb5kdc[22671](info): set up 2 sockets
Aug 12 14:52:49 dist-test-slave-6zbq krb5kdc[22671](info): commencing operation
krb5kdc: starting...
W20260812 14:52:50.619107 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:50 dist-test-slave-6zbq krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:52:50.726519 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:44331
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:50.804162 22702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:50.804347 22702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:50.804368 22702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:50.805490 22702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:52:50.805531 22702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:50.805544 22702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:50.805555 22702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:52:50.805567 22702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:52:50.807025 22702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:44331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:50.807251 22702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:50.807456 22702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:50.811976 22708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:50.812036 22702 server_base.cc:1034] running on GCE node
W20260812 14:52:50.812070 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:50.811964 22710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:50.812364 22702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:50.812606 22702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:50.813727 22702 hybrid_clock.cc:648] HybridClock initialized: now 1786546370813718 us; error 26 us; skew 500 ppm
Aug 12 14:52:50 dist-test-slave-6zbq krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546370, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:50.816078 22702 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:52:50.816437 22702 webserver.cc:460] Webserver started at http://127.4.27.62:46871/ using document root <none> and password file <none>
I20260812 14:52:50.816601 22702 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:50.816637 22702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:50.816735 22702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:50.817670 22702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "3d151acc46184481b4591aa7f025b152"
format_stamp: "Formatted at 2026-08-12 14:52:50 on dist-test-slave-6zbq"
server_key: "7101fd6953e0029006d0213a24cadc9e"
server_key_iv: "606384297de20a4e0af5e6690684b7fc"
server_key_version: "encryptionkey@0"
I20260812 14:52:50.817956 22702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "3d151acc46184481b4591aa7f025b152"
format_stamp: "Formatted at 2026-08-12 14:52:50 on dist-test-slave-6zbq"
server_key: "7101fd6953e0029006d0213a24cadc9e"
server_key_iv: "606384297de20a4e0af5e6690684b7fc"
server_key_version: "encryptionkey@0"
I20260812 14:52:50.819205 22702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:50.819878 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:50.820048 22702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:50.820130 22702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "3d151acc46184481b4591aa7f025b152"
format_stamp: "Formatted at 2026-08-12 14:52:50 on dist-test-slave-6zbq"
server_key: "7101fd6953e0029006d0213a24cadc9e"
server_key_iv: "606384297de20a4e0af5e6690684b7fc"
server_key_version: "encryptionkey@0"
I20260812 14:52:50.820181 22702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:50.827553 22702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:50.829365 22702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:50.829463 22702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:50.832978 22702 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:44331
I20260812 14:52:50.833019 22812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:44331 every 8 connection(s)
I20260812 14:52:50.833297 22702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:52:50.834168 22813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:50.836272 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152: Bootstrap starting.
I20260812 14:52:50.836794 22813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:50.837036 22813 log.cc:824] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:50.837651 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152: No bootstrap required, opened a new log
I20260812 14:52:50.839223 22813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d151acc46184481b4591aa7f025b152" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44331 } }
I20260812 14:52:50.839350 22813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:50.839383 22813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d151acc46184481b4591aa7f025b152, State: Initialized, Role: FOLLOWER
I20260812 14:52:50.839511 22813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d151acc46184481b4591aa7f025b152" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44331 } }
I20260812 14:52:50.839586 22813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:52:50.839618 22813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:52:50.839663 22813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:50.840379 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 22702
I20260812 14:52:50.840481 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:52:50.840461 22813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d151acc46184481b4591aa7f025b152" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44331 } }
I20260812 14:52:50.840554 22813 leader_election.cc:302] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d151acc46184481b4591aa7f025b152; no voters:
I20260812 14:52:50.840682 4204 external_mini_cluster.cc:1428] Setting key 5b2bd74379ca28ba2cfa0b100ee0f6b4
I20260812 14:52:50.840708 22813 leader_election.cc:288] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:52:50.840745 22818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:50.840911 22818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [term 1 LEADER]: Becoming Leader. State: Replica: 3d151acc46184481b4591aa7f025b152, State: Running, Role: LEADER
I20260812 14:52:50.840939 22813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:52:50.841099 22818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d151acc46184481b4591aa7f025b152" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44331 } }
I20260812 14:52:50.841521 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d151acc46184481b4591aa7f025b152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d151acc46184481b4591aa7f025b152" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44331 } } }
I20260812 14:52:50.841598 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:50.841810 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d151acc46184481b4591aa7f025b152. Latest consensus state: current_term: 1 leader_uuid: "3d151acc46184481b4591aa7f025b152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d151acc46184481b4591aa7f025b152" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44331 } } }
I20260812 14:52:50.842147 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152 [sys.catalog]: This master's current role is: LEADER
I20260812 14:52:50.842165 22838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:52:50.842808 22838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:52:50.844525 22838 catalog_manager.cc:1347] Generated new cluster ID: 5c99bd18ca6b41be9fef0b356efb29e6
I20260812 14:52:50.844627 22838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:52:50 dist-test-slave-6zbq krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:50.853417 22838 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:52:50.854112 22838 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:52:50.855525 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:50.841124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.4.27.62:44331)
0812 14:52:50.841267 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:50.841276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:50.841379 (+ 103us) server_negotiation.cc:407] Connection header received
0812 14:52:50.841499 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:50.841506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:50.841569 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:50.842872 (+ 1303us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:50.843730 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:50.845031 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:50.846042 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:50.846665 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:50.847014 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:50.849224 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:50.849228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:50.849228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:50.849239 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:50.853914 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:50.854321 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:50.854324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:50.854324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:50.854442 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:50.854625 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:50.854625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:50.854626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:50.854871 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:50.855003 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:50.855159 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 14:52:50.855300 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260812 14:52:50.857223 22838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152: Generated new TSK 0
I20260812 14:52:50.857355 22838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:52:50.885002 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44331
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:50.973803 22860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:50.973989 22860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:50.974017 22860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:50.975157 22860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:50.975198 22860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:50.975241 22860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:52:50.976619 22860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:50.976830 22860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:50.977023 22860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:50.982062 22871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:50.982070 22868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:50.982067 22867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:50.982437 22860 server_base.cc:1034] running on GCE node
I20260812 14:52:50.982607 22860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:50.982791 22860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:50.983918 22860 hybrid_clock.cc:648] HybridClock initialized: now 1786546370983907 us; error 27 us; skew 500 ppm
Aug 12 14:52:50 dist-test-slave-6zbq krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546370, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:50.986150 22860 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:52:50.986569 22860 webserver.cc:460] Webserver started at http://127.4.27.1:33113/ using document root <none> and password file <none>
I20260812 14:52:50.986732 22860 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:50.986770 22860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:50.986856 22860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:50.987752 22860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "007eb3ac504b4aa38881aa99a3cc19fc"
format_stamp: "Formatted at 2026-08-12 14:52:50 on dist-test-slave-6zbq"
server_key: "6afbd033a5b5ba7e747aadc6eb1317c4"
server_key_iv: "1e6fcb4fff983b255390487503604956"
server_key_version: "encryptionkey@0"
I20260812 14:52:50.988026 22860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "007eb3ac504b4aa38881aa99a3cc19fc"
format_stamp: "Formatted at 2026-08-12 14:52:50 on dist-test-slave-6zbq"
server_key: "6afbd033a5b5ba7e747aadc6eb1317c4"
server_key_iv: "1e6fcb4fff983b255390487503604956"
server_key_version: "encryptionkey@0"
I20260812 14:52:50.989117 22860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:52:50.989732 22892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:50.989866 22860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:50.989946 22860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "007eb3ac504b4aa38881aa99a3cc19fc"
format_stamp: "Formatted at 2026-08-12 14:52:50 on dist-test-slave-6zbq"
server_key: "6afbd033a5b5ba7e747aadc6eb1317c4"
server_key_iv: "1e6fcb4fff983b255390487503604956"
server_key_version: "encryptionkey@0"
I20260812 14:52:50.990005 22860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:51.005455 22860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:51.007081 22860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:51.007203 22860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:51.007390 22860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:51.007642 22860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:51.007675 22860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.007702 22860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:51.007727 22860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.013011 22860 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:32871
I20260812 14:52:51.013070 23108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:32871 every 8 connection(s)
I20260812 14:52:51.013331 22860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546370, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:51.019922 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 22860
I20260812 14:52:51.019994 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:52:51.020154 4204 external_mini_cluster.cc:1428] Setting key 40d1fa198f9f90545e5087ecc1393dee
I20260812 14:52:51.021445 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.013917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:52777 (local address 127.4.27.62:44331)
0812 14:52:51.013962 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.013964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.014133 (+ 169us) server_negotiation.cc:407] Connection header received
0812 14:52:51.014617 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.014617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.014631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.014681 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:51.015477 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.015961 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.016826 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.017004 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.017047 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.018853 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:51.018854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:51.018855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:51.018861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:51.020150 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.020556 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.020558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.020558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.020618 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.020856 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.020856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.020857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.020961 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:51.021012 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.021313 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.021382 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:52:51.021636 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.013886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:44331 (local address 127.4.27.1:52777)
0812 14:52:51.014030 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.014068 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.014535 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.014730 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.014734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.015242 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:51.015412 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.015416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.016020 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.016021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.016771 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.016773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.017056 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.017056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.017146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.017512 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:51.017521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:51.018777 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:51.020224 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:51.020239 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:51.020241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:51.020492 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:51.020681 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:51.020681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:51.020681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:51.020794 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:51.021021 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:51.021028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:51.021184 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.021296 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.021442 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260812 14:52:51.022089 23109 heartbeater.cc:347] Connected to a master server at 127.4.27.62:44331
I20260812 14:52:51.022177 23109 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:51.022364 23109 heartbeater.cc:511] Master 127.4.27.62:44331 requested a full tablet report, sending...
I20260812 14:52:51.022856 22754 ts_manager.cc:194] Registered new tserver with Master: 007eb3ac504b4aa38881aa99a3cc19fc (127.4.27.1:32871)
I20260812 14:52:51.023423 22754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:52777
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:52:51.043111 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44331
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:51.121429 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:51.121606 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:51.121627 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:51.122783 23115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:51.122838 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:51.122864 23115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:52:51.124274 23115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:51.124452 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:51.124612 23115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:51.129482 23125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:51.129499 23131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:51.129487 23127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:51.129528 23115 server_base.cc:1034] running on GCE node
I20260812 14:52:51.129882 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:51.130097 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:51.131230 23115 hybrid_clock.cc:648] HybridClock initialized: now 1786546371131199 us; error 47 us; skew 500 ppm
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:51.133437 23115 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:52:51.133833 23115 webserver.cc:460] Webserver started at http://127.4.27.2:34495/ using document root <none> and password file <none>
I20260812 14:52:51.134012 23115 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:51.134050 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:51.134138 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:51.135084 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "9e2c381768f9491daebe4e2614c318ed"
format_stamp: "Formatted at 2026-08-12 14:52:51 on dist-test-slave-6zbq"
server_key: "0b9bd24e79750906728d0278e6ccf7c5"
server_key_iv: "9ca1b7c704692a70f2daffedc6d823cd"
server_key_version: "encryptionkey@0"
I20260812 14:52:51.135370 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "9e2c381768f9491daebe4e2614c318ed"
format_stamp: "Formatted at 2026-08-12 14:52:51 on dist-test-slave-6zbq"
server_key: "0b9bd24e79750906728d0278e6ccf7c5"
server_key_iv: "9ca1b7c704692a70f2daffedc6d823cd"
server_key_version: "encryptionkey@0"
I20260812 14:52:51.136462 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:52:51.137065 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.137187 23115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:52:51.137262 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "9e2c381768f9491daebe4e2614c318ed"
format_stamp: "Formatted at 2026-08-12 14:52:51 on dist-test-slave-6zbq"
server_key: "0b9bd24e79750906728d0278e6ccf7c5"
server_key_iv: "9ca1b7c704692a70f2daffedc6d823cd"
server_key_version: "encryptionkey@0"
I20260812 14:52:51.137326 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:51.170956 23115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:51.172520 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:51.172626 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:51.172838 23115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:51.173123 23115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:51.173162 23115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.173182 23115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:51.173194 23115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.178520 23115 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:45645
I20260812 14:52:51.178560 23354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:45645 every 8 connection(s)
I20260812 14:52:51.178836 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:51.186596 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.179457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:44485 (local address 127.4.27.62:44331)
0812 14:52:51.179527 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.179529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.179665 (+ 136us) server_negotiation.cc:407] Connection header received
0812 14:52:51.180097 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.180098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.180109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.180156 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:51.180953 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.181453 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.182382 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.182590 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.182636 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.184505 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:51.184507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:51.184508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:51.184515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:51.185559 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.185881 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.185882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.185882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.185924 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.186097 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.186098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.186098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.186196 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:51.186241 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.186469 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.186534 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:52:51.186766 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.179417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:44331 (local address 127.4.27.2:44485)
0812 14:52:51.179553 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.179589 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.180014 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.180197 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.180201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.180715 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:51.180885 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.180889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.181509 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.181510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.182295 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.182297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.182646 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.182647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.182749 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.183184 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:51.183192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:51.184423 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:51.185621 (+ 1198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:51.185628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:51.185629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:51.185827 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:51.185966 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:51.185966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:51.185966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:51.186056 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:51.186238 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:51.186241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:51.186362 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.186453 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.186564 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260812 14:52:51.187224 23356 heartbeater.cc:347] Connected to a master server at 127.4.27.62:44331
I20260812 14:52:51.187323 23356 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:51.187408 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 23115
I20260812 14:52:51.187508 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:52:51.187505 23356 heartbeater.cc:511] Master 127.4.27.62:44331 requested a full tablet report, sending...
I20260812 14:52:51.187716 4204 external_mini_cluster.cc:1428] Setting key 21b1f864535f232c58a72852cce6ddef
I20260812 14:52:51.187863 22754 ts_manager.cc:194] Registered new tserver with Master: 9e2c381768f9491daebe4e2614c318ed (127.4.27.2:45645)
I20260812 14:52:51.188383 22754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:44485
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:52:51.210263 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44331
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:52:51.288038 23363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:52:51.288220 23363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:52:51.288241 23363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:52:51.289314 23363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:52:51.289346 23363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:52:51.289371 23363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:52:51.290737 23363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:52:51.290906 23363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:52:51.291069 23363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:52:51.295651 23374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:52:51.295632 23373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:51.295670 23363 server_base.cc:1034] running on GCE node
W20260812 14:52:51.295641 23378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:52:51.296007 23363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:52:51.296224 23363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:52:51.297362 23363 hybrid_clock.cc:648] HybridClock initialized: now 1786546371297351 us; error 28 us; skew 500 ppm
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:52:51.299675 23363 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:52:51.300072 23363 webserver.cc:460] Webserver started at http://127.4.27.3:40289/ using document root <none> and password file <none>
I20260812 14:52:51.300259 23363 fs_manager.cc:359] Metadata directory not provided
I20260812 14:52:51.300302 23363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:52:51.300414 23363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:52:51.301412 23363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "9360a09ad92f460b9ef57a57973fe910"
format_stamp: "Formatted at 2026-08-12 14:52:51 on dist-test-slave-6zbq"
server_key: "91a6a04e197291e15b58c317f4636f73"
server_key_iv: "34de43d563202adcea5061ea0f7e3a7b"
server_key_version: "encryptionkey@0"
I20260812 14:52:51.301726 23363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "9360a09ad92f460b9ef57a57973fe910"
format_stamp: "Formatted at 2026-08-12 14:52:51 on dist-test-slave-6zbq"
server_key: "91a6a04e197291e15b58c317f4636f73"
server_key_iv: "34de43d563202adcea5061ea0f7e3a7b"
server_key_version: "encryptionkey@0"
I20260812 14:52:51.302974 23363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:52:51.303625 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.303761 23363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:52:51.303821 23363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "9360a09ad92f460b9ef57a57973fe910"
format_stamp: "Formatted at 2026-08-12 14:52:51 on dist-test-slave-6zbq"
server_key: "91a6a04e197291e15b58c317f4636f73"
server_key_iv: "34de43d563202adcea5061ea0f7e3a7b"
server_key_version: "encryptionkey@0"
I20260812 14:52:51.303890 23363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:52:51.323421 23363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:52:51.325193 23363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:52:51.325300 23363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:52:51.325461 23363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:52:51.325716 23363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:52:51.325747 23363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.325768 23363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:52:51.325779 23363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:52:51.331318 23363 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:35435
I20260812 14:52:51.331380 23620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:35435 every 8 connection(s)
I20260812 14:52:51.331633 23363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:52:51.334457 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 23363
I20260812 14:52:51.334558 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:52:51.334743 4204 external_mini_cluster.cc:1428] Setting key bb8c8a643358bbcb7172e93dde494559
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546371, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:52:51.339838 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.332235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:45737 (local address 127.4.27.62:44331)
0812 14:52:51.332306 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.332308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.332478 (+ 170us) server_negotiation.cc:407] Connection header received
0812 14:52:51.332922 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.332923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.332933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.332980 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:51.333803 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.334289 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.335626 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.335818 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.335868 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.337756 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:51.337757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:51.337758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:51.337764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:51.338823 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.339136 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.339137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.339137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.339192 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.339359 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.339359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.339360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.339449 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:51.339492 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.339685 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.339775 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:52:51.339982 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.332220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:44331 (local address 127.4.27.3:45737)
0812 14:52:51.332374 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.332406 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.332853 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.333031 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.333034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.333546 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:52:51.333720 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.333725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.334386 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.334388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.335562 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.335565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.335875 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.335875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.335966 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.336396 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:52:51.336404 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:52:51.337681 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:52:51.338885 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:51.338890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:51.338892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:51.339083 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:51.339235 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:52:51.339235 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:52:51.339236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:52:51.339318 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:52:51.339489 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:52:51.339492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:52:51.339590 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.339664 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.339790 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260812 14:52:51.340363 23621 heartbeater.cc:347] Connected to a master server at 127.4.27.62:44331
I20260812 14:52:51.340442 23621 heartbeater.cc:464] Registering TS with master...
I20260812 14:52:51.340570 23621 heartbeater.cc:511] Master 127.4.27.62:44331 requested a full tablet report, sending...
I20260812 14:52:51.340898 22754 ts_manager.cc:194] Registered new tserver with Master: 9360a09ad92f460b9ef57a57973fe910 (127.4.27.3:35435)
I20260812 14:52:51.341320 22754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:45737
I20260812 14:52:51.346048 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:52:51.352695 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.347295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.4.27.62:44331)
0812 14:52:51.347332 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.347334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.347442 (+ 108us) server_negotiation.cc:407] Connection header received
0812 14:52:51.347498 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.347499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.347509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.347559 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:52:51.348080 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.348538 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.349219 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.349407 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.349460 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.350526 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:52:51.350528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:52:51.350528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:52:51.350535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:52:51.351705 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.351999 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.352000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.352000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.352051 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:52:51.352214 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:52:51.352215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:52:51.352216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:52:51.352353 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:52:51.352441 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.352543 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.352615 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:52:51.354053 22754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:52:51.355612 22754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:52:51.361373 23636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.357499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:35435 (local address 127.0.0.1:57786)
0812 14:52:51.357719 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.357727 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.357767 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.358114 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.358115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.358117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:51.358178 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.358180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.359206 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.359207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.360378 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.360381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.361094 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.361095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.361172 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.361185 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.361253 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.361306 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260812 14:52:51.361377 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.357640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.4.27.3:35435)
0812 14:52:51.357978 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.357982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.357989 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:52:51.358013 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.358015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.358025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.358089 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:51.358260 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.359119 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.360469 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.361010 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.361068 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.361136 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.361266 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.361304 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":72,"threads_started":1}
I20260812 14:52:51.361676 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.357504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:32871 (local address 127.0.0.1:55634)
0812 14:52:51.357851 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.357858 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.357889 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.358428 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.358429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.358431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:51.358465 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.358466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.359722 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.359723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.360535 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.360537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.361460 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.361461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.361521 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.361525 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.361554 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.361592 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":67,"threads_started":1}
I20260812 14:52:51.361738 23637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.357499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45645 (local address 127.0.0.1:53590)
0812 14:52:51.357652 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.357690 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.357755 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.358111 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.358113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.358115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:51.358196 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.358198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.359540 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.359541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.360592 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.360595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.361510 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.361511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.361563 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.361565 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.361614 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.361645 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":21,"thread_start_us":56,"threads_started":1}
I20260812 14:52:51.361770 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.357821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.4.27.2:45645)
0812 14:52:51.357951 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.357955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.357966 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:52:51.358005 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.358007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.358018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.358088 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:51.358289 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.359425 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.360655 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.361359 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.361429 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.361481 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.361647 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.361692 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260812 14:52:51.362032 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.357735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.4.27.1:32871)
0812 14:52:51.358221 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.358225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.358234 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:51.358292 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.358294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.358308 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.358537 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:51.358614 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.359629 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.360608 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.361359 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.361848 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.361918 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.361939 (+ 21us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.361970 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":72,"threads_started":1}
I20260812 14:52:51.362377 23484 tablet_service.cc:1467] Processing CreateTablet for tablet f4ae157c4f0647428b6f6f6b0ac617eb (DEFAULT_TABLE table=test-table [id=299da9e1e3b94ef283e598942108ce49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:51.362377 23236 tablet_service.cc:1467] Processing CreateTablet for tablet f4ae157c4f0647428b6f6f6b0ac617eb (DEFAULT_TABLE table=test-table [id=299da9e1e3b94ef283e598942108ce49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:51.362700 23484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ae157c4f0647428b6f6f6b0ac617eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:51.362789 23236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ae157c4f0647428b6f6f6b0ac617eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:51.362689 22982 tablet_service.cc:1467] Processing CreateTablet for tablet f4ae157c4f0647428b6f6f6b0ac617eb (DEFAULT_TABLE table=test-table [id=299da9e1e3b94ef283e598942108ce49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:52:51.363008 22982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ae157c4f0647428b6f6f6b0ac617eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:52:51.365864 23645 tablet_bootstrap.cc:492] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Bootstrap starting.
I20260812 14:52:51.366080 23648 tablet_bootstrap.cc:492] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Bootstrap starting.
I20260812 14:52:51.366387 23649 tablet_bootstrap.cc:492] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: Bootstrap starting.
I20260812 14:52:51.366796 23648 tablet_bootstrap.cc:654] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:51.367014 23649 tablet_bootstrap.cc:654] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:51.367358 23648 log.cc:824] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:51.367514 23649 log.cc:824] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:51.368113 23648 tablet_bootstrap.cc:492] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: No bootstrap required, opened a new log
I20260812 14:52:51.368170 23649 tablet_bootstrap.cc:492] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: No bootstrap required, opened a new log
I20260812 14:52:51.368175 23645 tablet_bootstrap.cc:654] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Neither blocks nor log segments found. Creating new log.
I20260812 14:52:51.368186 23648 ts_tablet_manager.cc:1397] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:52:51.368235 23649 ts_tablet_manager.cc:1397] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:52:51.368681 23645 log.cc:824] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Log is configured to *not* fsync() on all Append() calls
I20260812 14:52:51.369423 23645 tablet_bootstrap.cc:492] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: No bootstrap required, opened a new log
I20260812 14:52:51.369493 23645 ts_tablet_manager.cc:1397] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260812 14:52:51.369843 23649 raft_consensus.cc:348] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.369975 23649 raft_consensus.cc:374] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:51.370008 23649 raft_consensus.cc:729] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007eb3ac504b4aa38881aa99a3cc19fc, State: Initialized, Role: FOLLOWER
I20260812 14:52:51.370136 23649 consensus_queue.cc:260] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.370200 23648 raft_consensus.cc:348] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.370384 23648 raft_consensus.cc:374] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:51.370400 23649 ts_tablet_manager.cc:1428] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:52:51.370430 23648 raft_consensus.cc:729] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9360a09ad92f460b9ef57a57973fe910, State: Initialized, Role: FOLLOWER
I20260812 14:52:51.370507 23109 heartbeater.cc:503] Master 127.4.27.62:44331 was elected leader, sending a full tablet report...
I20260812 14:52:51.370539 23648 consensus_queue.cc:260] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.370813 23648 ts_tablet_manager.cc:1428] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:52:51.370870 23621 heartbeater.cc:503] Master 127.4.27.62:44331 was elected leader, sending a full tablet report...
I20260812 14:52:51.371351 23645 raft_consensus.cc:348] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.371486 23645 raft_consensus.cc:374] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:52:51.371520 23645 raft_consensus.cc:729] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2c381768f9491daebe4e2614c318ed, State: Initialized, Role: FOLLOWER
I20260812 14:52:51.371623 23645 consensus_queue.cc:260] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.371853 23645 ts_tablet_manager.cc:1428] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:52:51.371888 23356 heartbeater.cc:503] Master 127.4.27.62:44331 was elected leader, sending a full tablet report...
W20260812 14:52:51.429353 23358 tablet.cc:2369] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:52:51.513947 23110 tablet.cc:2369] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:52:51.514132 23665 raft_consensus.cc:482] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:52:51.514256 23665 raft_consensus.cc:504] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.514667 23665 leader_election.cc:288] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9360a09ad92f460b9ef57a57973fe910 (127.4.27.3:35435), 007eb3ac504b4aa38881aa99a3cc19fc (127.4.27.1:32871)
I20260812 14:52:51.517400 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.514976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:56139 (local address 127.4.27.3:35435)
0812 14:52:51.515019 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.515021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.515084 (+ 63us) server_negotiation.cc:407] Connection header received
0812 14:52:51.515159 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.515160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.515172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.515224 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:51.515369 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.515814 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.516706 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.517079 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.517137 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.517148 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.517288 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.517329 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:51.517416 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.514898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:50793 (local address 127.4.27.1:32871)
0812 14:52:51.514962 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.514964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.514975 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:52:51.515002 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.515004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.515016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.515082 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:52:51.515266 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.515769 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.516716 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.517086 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.517147 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.517158 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.517321 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.517360 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:52:51.517422 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.514821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:35435 (local address 127.4.27.2:56139)
0812 14:52:51.515034 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.515044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.515096 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.515241 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.515241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.515243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:51.515314 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.515315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.515886 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.515887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.516623 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.516624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.517151 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.517152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.517224 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.517228 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.517272 (+ 44us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.517311 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":52,"thread_start_us":93,"threads_started":1}
I20260812 14:52:51.517710 23523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ae157c4f0647428b6f6f6b0ac617eb" candidate_uuid: "9e2c381768f9491daebe4e2614c318ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9360a09ad92f460b9ef57a57973fe910" is_pre_election: true
I20260812 14:52:51.517426 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.514820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:32871 (local address 127.4.27.2:50793)
0812 14:52:51.514878 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.514891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:52:51.514939 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:52:51.515103 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:52:51.515104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:52:51.515105 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:52:51.515189 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.515191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.515839 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.515840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.516621 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:52:51.516624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.517167 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.517168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:52:51.517250 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.517253 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:52:51.517304 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:52:51.517342 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":17}
I20260812 14:52:51.517858 23523 raft_consensus.cc:2454] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2c381768f9491daebe4e2614c318ed in term 0.
I20260812 14:52:51.518117 23154 leader_election.cc:302] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9360a09ad92f460b9ef57a57973fe910, 9e2c381768f9491daebe4e2614c318ed; no voters:
I20260812 14:52:51.518134 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ae157c4f0647428b6f6f6b0ac617eb" candidate_uuid: "9e2c381768f9491daebe4e2614c318ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" is_pre_election: true
I20260812 14:52:51.518261 23015 raft_consensus.cc:2454] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2c381768f9491daebe4e2614c318ed in term 0.
I20260812 14:52:51.518280 23665 raft_consensus.cc:2781] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:52:51.518388 23665 raft_consensus.cc:482] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:52:51.518425 23665 raft_consensus.cc:3037] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:51.519102 23665 raft_consensus.cc:504] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.519222 23665 leader_election.cc:288] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [CANDIDATE]: Term 1 election: Requested vote from peers 9360a09ad92f460b9ef57a57973fe910 (127.4.27.3:35435), 007eb3ac504b4aa38881aa99a3cc19fc (127.4.27.1:32871)
I20260812 14:52:51.519356 23523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ae157c4f0647428b6f6f6b0ac617eb" candidate_uuid: "9e2c381768f9491daebe4e2614c318ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9360a09ad92f460b9ef57a57973fe910"
I20260812 14:52:51.519397 23015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ae157c4f0647428b6f6f6b0ac617eb" candidate_uuid: "9e2c381768f9491daebe4e2614c318ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "007eb3ac504b4aa38881aa99a3cc19fc"
I20260812 14:52:51.519428 23523 raft_consensus.cc:3037] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:51.519456 23015 raft_consensus.cc:3037] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:52:51.520109 23523 raft_consensus.cc:2454] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2c381768f9491daebe4e2614c318ed in term 1.
I20260812 14:52:51.520112 23015 raft_consensus.cc:2454] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2c381768f9491daebe4e2614c318ed in term 1.
I20260812 14:52:51.520274 23154 leader_election.cc:302] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9360a09ad92f460b9ef57a57973fe910, 9e2c381768f9491daebe4e2614c318ed; no voters:
I20260812 14:52:51.520349 23665 raft_consensus.cc:2781] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:52:51.520489 23665 raft_consensus.cc:686] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 1 LEADER]: Becoming Leader. State: Replica: 9e2c381768f9491daebe4e2614c318ed, State: Running, Role: LEADER
I20260812 14:52:51.520572 23665 consensus_queue.cc:237] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } }
I20260812 14:52:51.521359 22754 catalog_manager.cc:5495] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2c381768f9491daebe4e2614c318ed (127.4.27.2). New cstate: current_term: 1 leader_uuid: "9e2c381768f9491daebe4e2614c318ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e2c381768f9491daebe4e2614c318ed" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 } health_report { overall_health: UNKNOWN } } }
I20260812 14:52:51.540221 23641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.537919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.4.27.2:45645)
0812 14:52:51.537963 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:52:51.537965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:52:51.537974 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:52:51.538032 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:52:51.538033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:52:51.538053 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:52:51.538109 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:52:51.538237 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.538728 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.539488 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:52:51.539691 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:52:51.539743 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:52:51.539920 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:52:51.540014 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:52:51.540083 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:52:51.540121 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:52:51.540140 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:52:51.541803 23523 raft_consensus.cc:1261] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 1 FOLLOWER]: Refusing update from remote peer 9e2c381768f9491daebe4e2614c318ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:51.541841 23015 raft_consensus.cc:1261] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc [term 1 FOLLOWER]: Refusing update from remote peer 9e2c381768f9491daebe4e2614c318ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:52:51.542121 23671 consensus_queue.cc:1035] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9360a09ad92f460b9ef57a57973fe910" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:51.542263 23665 consensus_queue.cc:1035] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "007eb3ac504b4aa38881aa99a3cc19fc" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:52:51.544905 23680 mvcc.cc:205] Tried to move back new op lower bound from 7317693937832820736 to 7317693937748660224. Current Snapshot: MvccSnapshot[applied={T|T < 7317693937832820736}]
I20260812 14:52:51.544963 23684 mvcc.cc:205] Tried to move back new op lower bound from 7317693937832820736 to 7317693937748660224. Current Snapshot: MvccSnapshot[applied={T|T < 7317693937832820736}]
I20260812 14:52:51.547277 22754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:51.547358 22754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:52:51.548051 22754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d151acc46184481b4591aa7f025b152: Sending DeleteTablet for 3 replicas of tablet f4ae157c4f0647428b6f6f6b0ac617eb
I20260812 14:52:51.548369 23484 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ae157c4f0647428b6f6f6b0ac617eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:51 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:57786
I20260812 14:52:51.548391 23236 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ae157c4f0647428b6f6f6b0ac617eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:51 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:53590
I20260812 14:52:51.548592 22982 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ae157c4f0647428b6f6f6b0ac617eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:52:51 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260812 14:52:51.548611 23698 tablet_replica.cc:330] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: stopping tablet replica
I20260812 14:52:51.548718 23698 raft_consensus.cc:2229] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:52:51.548822 23698 raft_consensus.cc:2258] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:51.548838 23701 tablet_replica.cc:330] T f4ae157c4f0647428b6f6f6b0ac617eb P 007eb3ac504b4aa38881aa99a3cc19fc: stopping tablet replica
I20260812 14:52:51.549049 23694 tablet_replica.cc:330] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: stopping tablet replica
I20260812 14:52:51.549096 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 22860
I20260812 14:52:51.549136 23694 raft_consensus.cc:2229] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:52:51.549234 23694 raft_consensus.cc:2258] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:52:51.549602 23698 ts_tablet_manager.cc:1905] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:51.551299 23698 ts_tablet_manager.cc:1918] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:51.551375 23698 log.cc:1196] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/f4ae157c4f0647428b6f6f6b0ac617eb
I20260812 14:52:51.551687 23698 ts_tablet_manager.cc:1939] T f4ae157c4f0647428b6f6f6b0ac617eb P 9360a09ad92f460b9ef57a57973fe910: Deleting consensus metadata
I20260812 14:52:51.553798 23694 ts_tablet_manager.cc:1905] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:52:51.555296 23694 ts_tablet_manager.cc:1918] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:52:51.555358 23694 log.cc:1196] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/f4ae157c4f0647428b6f6f6b0ac617eb
I20260812 14:52:51.555606 23694 ts_tablet_manager.cc:1939] T f4ae157c4f0647428b6f6f6b0ac617eb P 9e2c381768f9491daebe4e2614c318ed: Deleting consensus metadata
I20260812 14:52:51.556088 22732 catalog_manager.cc:4841] TS 9e2c381768f9491daebe4e2614c318ed (127.4.27.2:45645): tablet f4ae157c4f0647428b6f6f6b0ac617eb (table test-table [id=299da9e1e3b94ef283e598942108ce49]) successfully deleted
I20260812 14:52:51.556658 22729 catalog_manager.cc:4841] TS 9360a09ad92f460b9ef57a57973fe910 (127.4.27.3:35435): tablet f4ae157c4f0647428b6f6f6b0ac617eb (table test-table [id=299da9e1e3b94ef283e598942108ce49]) successfully deleted
W20260812 14:52:51.561004 22734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:32871 (error 108)
I20260812 14:52:51.561472 23637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:52:51.561351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:32871 (local address 127.0.0.1:55636)
0812 14:52:51.561402 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:52:51.561441 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:52:51.561558 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 23115
W20260812 14:52:51.566936 22734 catalog_manager.cc:4584] TS 007eb3ac504b4aa38881aa99a3cc19fc (127.4.27.1:32871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4ae157c4f0647428b6f6f6b0ac617eb: Network error: Client connection negotiation failed: client connection to 127.4.27.1:32871: connect: Connection refused (error 111)
I20260812 14:52:51.571388 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 23363
I20260812 14:52:51.587440 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 22702
2026-08-12T14:52:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2158 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[23715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[23715](info): set up 2 sockets
Aug 12 14:52:51 dist-test-slave-6zbq krb5kdc[23715](info): commencing operation
krb5kdc: starting...
W20260812 14:52:52.769156 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:52:52 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:52:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/rangeradmin_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/rangeradmin_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/rangerlookup_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/rangerlookup_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
I20260812 14:52:52.914489 4204 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktWKW5U/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/postgres -l logfile start
2026-08-12 14:52:55.100 UTC [23769] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 14:52:55.100 UTC [23769] LOG: listening on IPv4 address "127.4.27.20", port 41487
2026-08-12 14:52:55.103 UTC [23769] LOG: listening on Unix socket "/tmp/.s.PGSQL.41487"
2026-08-12 14:52:55.131 UTC [23775] LOG: database system was shut down at 2026-08-12 14:52:53 UTC
2026-08-12 14:52:55.135 UTC [23769] LOG: database system is ready to accept connections
I20260812 14:52:56.229735 4204 mini_postgres.cc:96] Postgres bound to 41487
2026-08-12 14:52:56.237 UTC [23792] FATAL: database "slave" does not exist
127.4.27.20:41487 - accepting connections
I20260812 14:52:56.238075 4204 mini_ranger.cc:161] Starting Ranger...
I20260812 14:52:56.252568 4204 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 14:52:56.640400 4204 mini_ranger.cc:87] Created ranger Postgres database
I20260812 14:52:56.640468 4204 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-admin
2026-08-12 14:52:57,018 [I] DB FLAVOR :POSTGRES
2026-08-12 14:52:57,019 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:52:57,019 [I] Checking connection..
2026-08-12 14:52:57,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 14:52:57,285 [I] Checking connection passed.
2026-08-12 14:52:57,286 [I] --------- Verifying version history table ---------
2026-08-12 14:52:57,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:52:57,550 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:52:57,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:52:57,786 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:52:57,787 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 14:52:57,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 14:52:58.033 UTC [23924] WARNING: there is no transaction in progress
2026-08-12 14:52:58,041 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 14:52:58,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:52:58,277 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 14:52:58,278 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:52:58,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 14:52:58,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 14:52:58,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6zbq.c.gcp-upstream.internal','N') ;"
2026-08-12 14:52:59,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:52:59,257 [I] Table x_portal_user does not exist in database ranger
2026-08-12 14:52:59,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:52:59,512 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 14:52:59,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:52:59,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:53:00,006 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 14:53:00,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 14:53:00.874 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:00.892 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:00.905 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:00.918 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:00.931 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:00.946 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.005 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.010 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.022 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.031 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.040 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.053 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.061 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.067 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.072 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01.077 UTC [24121] WARNING: there is no transaction in progress
2026-08-12 14:53:01,608 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 14:53:01,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 14:53:01,852 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 14:53:01,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:53:02,090 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 14:53:02,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:53:02,328 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 14:53:02,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:53:02,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:53:02,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6zbq.c.gcp-upstream.internal';"
2026-08-12 14:53:03,048 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 14:53:03,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 14:53:03,296 [I] DB_PATCHES have already been applied
I20260812 14:53:03.302428 4204 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktWKW5U/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 14:53:03.302488 4204 mini_ranger.cc:193] Using host: 127.4.27.20
I20260812 14:53:03.305995 4204 mini_ranger.cc:227] Ranger admin URL: http://127.4.27.20:44561
Aug 12, 2026 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:53:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:53:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 14:53:04.651793 4204 mini_ranger.cc:160] Time spent starting Ranger: real 8.414s user 0.001s sys 0.008s
Aug 12, 2026 2:53:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:53:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:53:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:53:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:53:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 14:53:24.769277 4204 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/rangerkms_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/rangerkms_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.27.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.27.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 14:53:24 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546404, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 14:53:24.841985 4204 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 14:53:24.850248 4204 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 14:53:25.517112 4204 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 14:53:25.517181 4204 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms
I20260812 14:53:25.517198 4204 mini_ranger_kms.cc:227] Using postgres at 127.4.27.20:41487
2026-08-12 14:53:25,770 [I] DB FLAVOR :POSTGRES
2026-08-12 14:53:25,770 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:53:25,770 [I] Checking connection
2026-08-12 14:53:25,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 14:53:26,011 [I] connection success
2026-08-12 14:53:26,011 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 14:53:26,011 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 14:53:26,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 14:53:26,231 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 14:53:26,231 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:53:26,232 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 14:53:26,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:41487/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 14:53:26,473 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 14:53:26.889312 4204 mini_ranger_kms.cc:310] Created kms service
I20260812 14:53:27.063953 4204 mini_ranger_kms.cc:326] Created kudu user
I20260812 14:53:27.103613 4204 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 14:53:27 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546404, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:53:27.468569 4204 mini_ranger_kms.cc:384] Added ranger policy
I20260812 14:53:27.468703 4204 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 14:53:27.468732 4204 mini_ranger_kms.cc:240] Using host: 127.4.27.20
I20260812 14:53:27.473191 4204 mini_ranger_kms.cc:276] Ranger KMS PID: 24442
I20260812 14:53:27.473299 4204 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.27.20:52545
Aug 12, 2026 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 14:53:29 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786546409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.27.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:53:30.101770 4204 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.260s user 0.004s sys 0.231s
I20260812 14:53:30.101866 4204 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 14:53:30.101948 4204 mini_ranger_kms.cc:401] 127.4.27.20:52545/kms/v1/keys
Aug 12 14:53:30 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.27.20@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
Aug 12 14:53:30 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.27.20@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
Aug 12 14:53:32 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:53:32.866535 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:38955
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:38955
--ranger_config_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:52545/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:53:32.948980 24550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:53:32.949174 24550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:53:32.949195 24550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:53:32.950304 24550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:53:32.950364 24550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:53:32.950387 24550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:53:32.950416 24550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:53:32.950439 24550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:53:32.952059 24550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.27.20:52545/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:38955
--ranger_config_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:38955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:53:32.952273 24550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:53:32.952474 24550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:53:32.958194 24561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:32.958185 24559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:32.958184 24558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:53:32.958561 24550 server_base.cc:1034] running on GCE node
I20260812 14:53:32.958809 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:53:32.959061 24550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:53:32.960196 24550 hybrid_clock.cc:648] HybridClock initialized: now 1786546412960173 us; error 40 us; skew 500 ppm
Aug 12 14:53:32 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546412, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:53:32.966820 24550 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:53:32.967293 24550 webserver.cc:460] Webserver started at http://127.4.27.62:38233/ using document root <none> and password file <none>
I20260812 14:53:32.967511 24550 fs_manager.cc:359] Metadata directory not provided
I20260812 14:53:32.967558 24550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:53:32.967682 24550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:53:32 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546412, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:53:33.011390 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "1e939e48162c4d11a1ef5aac8488eaef"
format_stamp: "Formatted at 2026-08-12 14:53:33 on dist-test-slave-6zbq"
server_key: "c77939a98142e6349e0b268bec8c0288"
server_key_iv: "86f75126acda59f649031a47e0e3cbe6"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:33.011775 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "1e939e48162c4d11a1ef5aac8488eaef"
format_stamp: "Formatted at 2026-08-12 14:53:33 on dist-test-slave-6zbq"
server_key: "c77939a98142e6349e0b268bec8c0288"
server_key_iv: "86f75126acda59f649031a47e0e3cbe6"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:33.013175 24550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:53:33.027998 24571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:33.028198 24550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:53:33.028291 24550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "1e939e48162c4d11a1ef5aac8488eaef"
format_stamp: "Formatted at 2026-08-12 14:53:33 on dist-test-slave-6zbq"
server_key: "c77939a98142e6349e0b268bec8c0288"
server_key_iv: "86f75126acda59f649031a47e0e3cbe6"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:33.028367 24550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:53:33.041643 24550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:53:33.043362 24550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:53:33.043481 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:53:33.047603 24550 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:38955
I20260812 14:53:33.047680 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:38955 every 8 connection(s)
I20260812 14:53:33.047966 24550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:53:33.048695 24658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:53:33.051896 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 24550
I20260812 14:53:33.051957 24658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef: Bootstrap starting.
I20260812 14:53:33.051994 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:53:33.052193 4204 external_mini_cluster.cc:1428] Setting key ed531383ab68cc1eb4210ca1c6a628a2
I20260812 14:53:33.052757 24658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef: Neither blocks nor log segments found. Creating new log.
I20260812 14:53:33.053149 24658 log.cc:824] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef: Log is configured to *not* fsync() on all Append() calls
I20260812 14:53:33.053911 24658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef: No bootstrap required, opened a new log
I20260812 14:53:33.055884 24658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e939e48162c4d11a1ef5aac8488eaef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38955 } }
I20260812 14:53:33.056046 24658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:53:33.056075 24658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e939e48162c4d11a1ef5aac8488eaef, State: Initialized, Role: FOLLOWER
I20260812 14:53:33.056228 24658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e939e48162c4d11a1ef5aac8488eaef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38955 } }
I20260812 14:53:33.056346 24658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:53:33.056398 24658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:53:33.056449 24658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:53:33.056968 24658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e939e48162c4d11a1ef5aac8488eaef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38955 } }
I20260812 14:53:33.057067 24658 leader_election.cc:302] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e939e48162c4d11a1ef5aac8488eaef; no voters:
I20260812 14:53:33.057226 24658 leader_election.cc:288] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:53:33.057542 24658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:53:33.057508 24667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e939e48162c4d11a1ef5aac8488eaef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38955 } } }
I20260812 14:53:33.057761 24667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:53:33.057922 24665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:53:33.058320 24665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [term 1 LEADER]: Becoming Leader. State: Replica: 1e939e48162c4d11a1ef5aac8488eaef, State: Running, Role: LEADER
I20260812 14:53:33.059620 24665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e939e48162c4d11a1ef5aac8488eaef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38955 } }
Aug 12 14:53:33 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:53:33.060134 24667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e939e48162c4d11a1ef5aac8488eaef. Latest consensus state: current_term: 1 leader_uuid: "1e939e48162c4d11a1ef5aac8488eaef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e939e48162c4d11a1ef5aac8488eaef" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 38955 } } }
I20260812 14:53:33.060199 24667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef [sys.catalog]: This master's current role is: LEADER
I20260812 14:53:33.060458 24684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:53:33.060909 24658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:53:33.061544 24684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:53:33.067386 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:33.052660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.4.27.62:38955)
0812 14:53:33.052843 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:33.052852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:33.052884 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:53:33.053384 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:33.053394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:33.053458 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:33.053615 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:53:33.054401 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:33.056204 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:33.057301 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:33.057902 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:33.058010 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:33.060712 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:53:33.060716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:53:33.060716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:53:33.060728 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:53:33.064794 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:33.065234 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:33.065241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:33.065242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:33.065426 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:33.065645 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:33.065646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:33.065646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:33.066231 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:53:33.066473 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:33.066558 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 14:53:33.067004 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260812 14:53:33.068142 24684 catalog_manager.cc:1347] Generated new cluster ID: 5b2ce9f1cba14a46bd77f16efeb84ca8
I20260812 14:53:33.068190 24684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:53:33.077404 24684 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:53:33.077893 24684 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:53:33.088696 24684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef: Generated new TSK 0
I20260812 14:53:33.088872 24684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:53:33.751299 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:52545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38955
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:53:33.838400 24736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:53:33.838583 24736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:53:33.838622 24736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:53:33.839756 24736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:53:33.839797 24736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:53:33.839830 24736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:53:33.841295 24736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.27.20:52545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:53:33.841511 24736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:53:33.841734 24736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:53:33.846482 24746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:33.846594 24750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:33.846482 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:53:33.847223 24736 server_base.cc:1034] running on GCE node
I20260812 14:53:33.847424 24736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:53:33.847651 24736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:53:33.848802 24736 hybrid_clock.cc:648] HybridClock initialized: now 1786546413848791 us; error 27 us; skew 500 ppm
Aug 12 14:53:33 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546413, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:53:33.851167 24736 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:53:33.851526 24736 webserver.cc:460] Webserver started at http://127.4.27.1:41857/ using document root <none> and password file <none>
I20260812 14:53:33.851738 24736 fs_manager.cc:359] Metadata directory not provided
I20260812 14:53:33.851787 24736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:53:33.851908 24736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:53:33 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546413, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:53:33.874893 24736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "17642f7fd20a4e0488e0911bff04ab75"
format_stamp: "Formatted at 2026-08-12 14:53:33 on dist-test-slave-6zbq"
server_key: "2e059c8529ca5a6dffd1b073047accfd"
server_key_iv: "efaf7fefe9304cc6de5ff54e590dbd87"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:33.875456 24736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "17642f7fd20a4e0488e0911bff04ab75"
format_stamp: "Formatted at 2026-08-12 14:53:33 on dist-test-slave-6zbq"
server_key: "2e059c8529ca5a6dffd1b073047accfd"
server_key_iv: "efaf7fefe9304cc6de5ff54e590dbd87"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:33.877647 24736 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 14:53:33.893234 24762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:33.893457 24736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:53:33.893560 24736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "17642f7fd20a4e0488e0911bff04ab75"
format_stamp: "Formatted at 2026-08-12 14:53:33 on dist-test-slave-6zbq"
server_key: "2e059c8529ca5a6dffd1b073047accfd"
server_key_iv: "efaf7fefe9304cc6de5ff54e590dbd87"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:33.893666 24736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:53:33.922726 24736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:53:33.925100 24736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:53:33.925300 24736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:53:33.925603 24736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:53:33.926043 24736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:53:33.926142 24736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:33.926251 24736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:53:33.926335 24736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:33.933233 24736 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:42071
I20260812 14:53:33.933727 24736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:53:33.936368 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:42071 every 8 connection(s)
I20260812 14:53:33.939666 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 24736
I20260812 14:53:33.939767 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:53:33.939965 4204 external_mini_cluster.cc:1428] Setting key 042fb6af03e07047d5fb9a592e50e6d7
Aug 12 14:53:33 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546413, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:53:33.952152 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:33.937018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38955 (local address 127.4.27.1:60485)
0812 14:53:33.939584 (+ 2566us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:33.939634 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:33.940270 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:33.942723 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:33.942728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:33.942958 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:53:33.943232 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:33.943238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:33.944232 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:33.944233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:33.945546 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:33.945550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:33.946172 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:33.946173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:33.946325 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:33.946895 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:53:33.946905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:53:33.948474 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:53:33.950467 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:53:33.950475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:53:33.950478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:53:33.950787 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:53:33.951011 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:53:33.951012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:53:33.951012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:53:33.951147 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:53:33.951455 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:53:33.951462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:53:33.951655 (+ 193us) client_negotiation.cc:769] Sending connection context
0812 14:53:33.951768 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:53:33.951922 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2508,"thread_start_us":83,"threads_started":1}
I20260812 14:53:33.952689 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:33.937044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:60485 (local address 127.4.27.62:38955)
0812 14:53:33.942452 (+ 5408us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:33.942454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:33.942463 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:53:33.942487 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:33.942488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:33.942500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:33.942571 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:53:33.943341 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:33.944124 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:33.945663 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:33.945988 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:33.946098 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:33.948602 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:53:33.948603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:53:33.948604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:53:33.948611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:53:33.950375 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:33.950874 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:33.950875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:33.950876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:33.950943 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:33.951223 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:33.951224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:33.951225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:33.951366 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:53:33.951427 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:33.952495 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0812 14:53:33.952580 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5374,"thread_start_us":90,"threads_started":1}
I20260812 14:53:33.953488 24964 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38955
I20260812 14:53:33.953593 24964 heartbeater.cc:464] Registering TS with master...
I20260812 14:53:33.953773 24964 heartbeater.cc:511] Master 127.4.27.62:38955 requested a full tablet report, sending...
I20260812 14:53:33.954443 24604 ts_manager.cc:194] Registered new tserver with Master: 17642f7fd20a4e0488e0911bff04ab75 (127.4.27.1:42071)
I20260812 14:53:33.955349 24604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:60485
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:53:33.982822 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:52545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38955
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:53:34.100878 24971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:53:34.101064 24971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:53:34.101118 24971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:53:34.102838 24971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:53:34.102949 24971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:53:34.103035 24971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:53:34.105224 24971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.27.20:52545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:53:34.105641 24971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:53:34.105954 24971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:53:34.111061 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:34.111371 24987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:53:34.111475 24971 server_base.cc:1034] running on GCE node
W20260812 14:53:34.111640 24992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:53:34.111824 24971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:53:34.112051 24971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:53:34.113207 24971 hybrid_clock.cc:648] HybridClock initialized: now 1786546414113194 us; error 28 us; skew 500 ppm
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:53:34.116219 24971 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:53:34.116710 24971 webserver.cc:460] Webserver started at http://127.4.27.2:32975/ using document root <none> and password file <none>
I20260812 14:53:34.116976 24971 fs_manager.cc:359] Metadata directory not provided
I20260812 14:53:34.117033 24971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:53:34.117162 24971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:53:34.131384 24971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304"
format_stamp: "Formatted at 2026-08-12 14:53:34 on dist-test-slave-6zbq"
server_key: "3273c031316e2af337b4e41dabca8be7"
server_key_iv: "54c1386e24c23ae9fb4f06a9832f3246"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:34.131819 24971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304"
format_stamp: "Formatted at 2026-08-12 14:53:34 on dist-test-slave-6zbq"
server_key: "3273c031316e2af337b4e41dabca8be7"
server_key_iv: "54c1386e24c23ae9fb4f06a9832f3246"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:34.139545 24971 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260812 14:53:34.163434 25006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:34.163656 24971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 14:53:34.163738 24971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304"
format_stamp: "Formatted at 2026-08-12 14:53:34 on dist-test-slave-6zbq"
server_key: "3273c031316e2af337b4e41dabca8be7"
server_key_iv: "54c1386e24c23ae9fb4f06a9832f3246"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:34.163823 24971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:53:34.203184 24971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:53:34.205431 24971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:53:34.205617 24971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:53:34.205912 24971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:53:34.206383 24971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:53:34.206468 24971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:34.206534 24971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:53:34.206583 24971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:34.214130 24971 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36401
I20260812 14:53:34.214632 24971 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:53:34.215220 25214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36401 every 8 connection(s)
I20260812 14:53:34.216302 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 24971
I20260812 14:53:34.216425 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:53:34.216631 4204 external_mini_cluster.cc:1428] Setting key 1859ea1b1b4400d91d9ece3781e0a1cd
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:53:34.226423 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.215698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:50283 (local address 127.4.27.62:38955)
0812 14:53:34.215746 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.215748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.215813 (+ 65us) server_negotiation.cc:407] Connection header received
0812 14:53:34.216634 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.216636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.216652 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.216682 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:53:34.217963 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.218736 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.220188 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.220520 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.220564 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.223123 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:53:34.223125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:53:34.223126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:53:34.223134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:53:34.224878 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:34.225350 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:34.225352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:34.225352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:34.225420 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:34.225681 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:34.225682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:34.225682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:34.225839 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:53:34.225909 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.226210 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.226306 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:53:34.226879 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.215402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38955 (local address 127.4.27.2:50283)
0812 14:53:34.215576 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.215620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.216504 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.217320 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.217324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.217567 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:53:34.217847 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.217854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.218857 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.218857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.220085 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.220089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.220666 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.220667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.220803 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.221357 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:53:34.221367 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:53:34.223012 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:53:34.224972 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:53:34.224979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:53:34.224982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:53:34.225281 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:53:34.225493 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:53:34.225494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:53:34.225494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:53:34.225624 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:53:34.225899 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:53:34.225905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:53:34.226073 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.226186 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.226373 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260812 14:53:34.227525 25215 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38955
I20260812 14:53:34.227631 25215 heartbeater.cc:464] Registering TS with master...
I20260812 14:53:34.227815 25215 heartbeater.cc:511] Master 127.4.27.62:38955 requested a full tablet report, sending...
I20260812 14:53:34.228179 24604 ts_manager.cc:194] Registered new tserver with Master: 5d5da0ea20f34d59bfdbe1e0b3bbd304 (127.4.27.2:36401)
I20260812 14:53:34.228674 24604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:50283
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:53:34.253068 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:52545/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38955
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:53:34.369681 25222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:53:34.369949 25222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:53:34.369989 25222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:53:34.371663 25222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:53:34.371714 25222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:53:34.371752 25222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:53:34.373929 25222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.27.20:52545/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:38955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:53:34.374218 25222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:53:34.374523 25222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:53:34.381131 25236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:34.381486 25239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:53:34.381698 25235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:53:34.382823 25222 server_base.cc:1034] running on GCE node
I20260812 14:53:34.383075 25222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:53:34.383405 25222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:53:34.384650 25222 hybrid_clock.cc:648] HybridClock initialized: now 1786546414384627 us; error 34 us; skew 500 ppm
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:53:34.387998 25222 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:53:34.388512 25222 webserver.cc:460] Webserver started at http://127.4.27.3:35401/ using document root <none> and password file <none>
I20260812 14:53:34.388767 25222 fs_manager.cc:359] Metadata directory not provided
I20260812 14:53:34.388820 25222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:53:34.388945 25222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:53:34.406167 25222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "ecdf92a4a20e4782b0995d259e491876"
format_stamp: "Formatted at 2026-08-12 14:53:34 on dist-test-slave-6zbq"
server_key: "ab6c5f4b92866642a7a200f4d8760b09"
server_key_iv: "b5b96ef0cd5e9d9c59e9fa2c423d3e96"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:34.406714 25222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "ecdf92a4a20e4782b0995d259e491876"
format_stamp: "Formatted at 2026-08-12 14:53:34 on dist-test-slave-6zbq"
server_key: "ab6c5f4b92866642a7a200f4d8760b09"
server_key_iv: "b5b96ef0cd5e9d9c59e9fa2c423d3e96"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:34.408540 25222 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 14:53:34.420248 25251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:34.420398 25222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:53:34.420485 25222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "ecdf92a4a20e4782b0995d259e491876"
format_stamp: "Formatted at 2026-08-12 14:53:34 on dist-test-slave-6zbq"
server_key: "ab6c5f4b92866642a7a200f4d8760b09"
server_key_iv: "b5b96ef0cd5e9d9c59e9fa2c423d3e96"
server_key_version: "kuduclusterkey@0"
I20260812 14:53:34.420567 25222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:53:34.434207 25222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:53:34.436401 25222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:53:34.436576 25222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:53:34.436792 25222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:53:34.437175 25222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:53:34.437230 25222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:34.437299 25222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:53:34.437325 25222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:53:34.443651 25222 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:36321
I20260812 14:53:34.444144 25222 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:53:34.449816 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 25222
I20260812 14:53:34.449930 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:53:34.450158 4204 external_mini_cluster.cc:1428] Setting key 81467561b8ac4c688d882adef25c2123
I20260812 14:53:34.454742 25456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:36321 every 8 connection(s)
Aug 12 14:53:34 dist-test-slave-6zbq krb5kdc[23715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:53:34.468576 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.455602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:57079 (local address 127.4.27.62:38955)
0812 14:53:34.455673 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.455675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.455714 (+ 39us) server_negotiation.cc:407] Connection header received
0812 14:53:34.456437 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.456438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.456451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.456481 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:53:34.457357 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.458112 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.459660 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.459964 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.460030 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.465360 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:53:34.465362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:53:34.465363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:53:34.465371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:53:34.467072 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:34.467544 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:34.467545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:34.467546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:34.467608 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:34.467865 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:34.467865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:34.467866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:34.468001 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:53:34.468071 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.468385 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.468475 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 14:53:34.468745 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.455296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:38955 (local address 127.4.27.3:57079)
0812 14:53:34.455491 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.455549 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.456307 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.456681 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.456686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.456941 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:53:34.457245 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.457252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.458236 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.458237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.459546 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.459550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.460060 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.460061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.460226 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.460834 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:53:34.460845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:53:34.465145 (+ 4300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:53:34.467173 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:53:34.467180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:53:34.467183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:53:34.467469 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:53:34.467672 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:53:34.467673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:53:34.467673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:53:34.467802 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:53:34.468064 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:53:34.468068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:53:34.468240 (+ 172us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.468351 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.468504 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260812 14:53:34.469425 25460 heartbeater.cc:347] Connected to a master server at 127.4.27.62:38955
I20260812 14:53:34.469539 25460 heartbeater.cc:464] Registering TS with master...
I20260812 14:53:34.469744 25460 heartbeater.cc:511] Master 127.4.27.62:38955 requested a full tablet report, sending...
I20260812 14:53:34.470162 24604 ts_manager.cc:194] Registered new tserver with Master: ecdf92a4a20e4782b0995d259e491876 (127.4.27.3:36321)
I20260812 14:53:34.470789 24604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:57079
I20260812 14:53:34.475653 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:53:34.484983 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.477393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.4.27.62:38955)
0812 14:53:34.477429 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.477431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.477688 (+ 257us) server_negotiation.cc:407] Connection header received
0812 14:53:34.477774 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.477775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.477785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.477810 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:53:34.478541 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.479261 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.480306 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.480585 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.480631 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.482016 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:53:34.482018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:53:34.482018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:53:34.482026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:53:34.483704 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:34.484127 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:34.484129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:34.484129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:34.484189 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:53:34.484393 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:53:34.484394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:53:34.484394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:53:34.484517 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:53:34.484594 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.484815 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.484892 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:53:34.486876 24604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:53:34.489233 24604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:53:34.498402 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.492563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36401 (local address 127.0.0.1:45318)
0812 14:53:34.492914 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.492955 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.493020 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.494108 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.494112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.494115 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:53:34.494200 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.494202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.495536 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.495537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.497060 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.497063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.498111 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.498112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.498192 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.498205 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.498267 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.498315 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":74,"threads_started":1}
I20260812 14:53:34.498724 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.493300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.4.27.2:36401)
0812 14:53:34.493923 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.493927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.493939 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:53:34.493979 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.493981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.493996 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.494075 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:53:34.494310 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.495395 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.497186 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.497925 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.497992 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.498074 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.498589 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.498646 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":90,"threads_started":1}
I20260812 14:53:34.499173 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.492775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:42071 (local address 127.0.0.1:56342)
0812 14:53:34.494197 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.494207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.494386 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.494709 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.494710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.494713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:53:34.494757 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.494758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.496235 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.496236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.497265 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.497269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.498728 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.498729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.499004 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.499010 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.499059 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.499101 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":68,"threads_started":1}
I20260812 14:53:34.499447 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.493607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.4.27.1:42071)
0812 14:53:34.493781 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.493786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.494438 (+ 652us) server_negotiation.cc:407] Connection header received
0812 14:53:34.494477 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.494479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.494495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.494559 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:53:34.494847 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.496103 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.497392 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.498575 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.498640 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.498697 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.499332 (+ 635us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.499379 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260812 14:53:34.499411 25104 tablet_service.cc:1467] Processing CreateTablet for tablet 1212b0feac664c5680ad42723f29c845 (DEFAULT_TABLE table=test-table [id=30403fbd0a5144539f23a54b0ceed528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:53:34.499785 25104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1212b0feac664c5680ad42723f29c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:53:34.500023 24841 tablet_service.cc:1467] Processing CreateTablet for tablet 1212b0feac664c5680ad42723f29c845 (DEFAULT_TABLE table=test-table [id=30403fbd0a5144539f23a54b0ceed528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:53:34.500363 24841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1212b0feac664c5680ad42723f29c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:53:34.503845 25495 tablet_bootstrap.cc:492] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75: Bootstrap starting.
I20260812 14:53:34.504545 25495 tablet_bootstrap.cc:654] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75: Neither blocks nor log segments found. Creating new log.
I20260812 14:53:34.504865 25495 log.cc:824] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75: Log is configured to *not* fsync() on all Append() calls
I20260812 14:53:34.506527 25494 tablet_bootstrap.cc:492] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Bootstrap starting.
I20260812 14:53:34.507560 25494 tablet_bootstrap.cc:654] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Neither blocks nor log segments found. Creating new log.
I20260812 14:53:34.507893 25494 log.cc:824] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Log is configured to *not* fsync() on all Append() calls
I20260812 14:53:34.507987 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.492913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:36321 (local address 127.0.0.1:58716)
0812 14:53:34.498475 (+ 5562us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.498484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.498518 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.504480 (+ 5962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.504481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.504483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:53:34.504515 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.504517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.505702 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.505704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.506780 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.506783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.507774 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.507775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.507834 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.507838 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.507883 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.507923 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5552,"thread_start_us":54,"threads_started":1}
I20260812 14:53:34.508252 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.493279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.4.27.3:36321)
0812 14:53:34.504313 (+ 11034us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.504317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.504327 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:53:34.504363 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.504365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.504379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.504467 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:53:34.504612 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.505611 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.506896 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.507585 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.507667 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.507741 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.508132 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.508177 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10996,"thread_start_us":83,"threads_started":1}
I20260812 14:53:34.508924 25495 tablet_bootstrap.cc:492] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75: No bootstrap required, opened a new log
I20260812 14:53:34.508862 25324 tablet_service.cc:1467] Processing CreateTablet for tablet 1212b0feac664c5680ad42723f29c845 (DEFAULT_TABLE table=test-table [id=30403fbd0a5144539f23a54b0ceed528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:53:34.508997 25495 ts_tablet_manager.cc:1397] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 14:53:34.509153 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1212b0feac664c5680ad42723f29c845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:53:34.511384 25495 raft_consensus.cc:348] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.511546 25495 raft_consensus.cc:374] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:53:34.511584 25495 raft_consensus.cc:729] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17642f7fd20a4e0488e0911bff04ab75, State: Initialized, Role: FOLLOWER
I20260812 14:53:34.511699 25495 consensus_queue.cc:260] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.511940 25495 ts_tablet_manager.cc:1428] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:53:34.512053 24964 heartbeater.cc:503] Master 127.4.27.62:38955 was elected leader, sending a full tablet report...
I20260812 14:53:34.512171 25500 tablet_bootstrap.cc:492] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Bootstrap starting.
I20260812 14:53:34.513224 25500 tablet_bootstrap.cc:654] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Neither blocks nor log segments found. Creating new log.
I20260812 14:53:34.513504 25500 log.cc:824] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Log is configured to *not* fsync() on all Append() calls
I20260812 14:53:34.513936 25494 tablet_bootstrap.cc:492] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: No bootstrap required, opened a new log
I20260812 14:53:34.514014 25494 ts_tablet_manager.cc:1397] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260812 14:53:34.515954 25494 raft_consensus.cc:348] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.516134 25494 raft_consensus.cc:374] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:53:34.516167 25494 raft_consensus.cc:729] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5da0ea20f34d59bfdbe1e0b3bbd304, State: Initialized, Role: FOLLOWER
I20260812 14:53:34.516279 25494 consensus_queue.cc:260] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.516526 25494 ts_tablet_manager.cc:1428] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:53:34.516577 25215 heartbeater.cc:503] Master 127.4.27.62:38955 was elected leader, sending a full tablet report...
I20260812 14:53:34.518992 25500 tablet_bootstrap.cc:492] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: No bootstrap required, opened a new log
I20260812 14:53:34.519062 25500 ts_tablet_manager.cc:1397] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 14:53:34.520896 25500 raft_consensus.cc:348] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.521046 25500 raft_consensus.cc:374] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:53:34.521075 25500 raft_consensus.cc:729] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdf92a4a20e4782b0995d259e491876, State: Initialized, Role: FOLLOWER
I20260812 14:53:34.521188 25500 consensus_queue.cc:260] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.521427 25500 ts_tablet_manager.cc:1428] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:53:34.521503 25460 heartbeater.cc:503] Master 127.4.27.62:38955 was elected leader, sending a full tablet report...
I20260812 14:53:34.546120 25499 raft_consensus.cc:482] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:53:34.546277 25499 raft_consensus.cc:504] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.547506 25499 leader_election.cc:288] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d5da0ea20f34d59bfdbe1e0b3bbd304 (127.4.27.2:36401), ecdf92a4a20e4782b0995d259e491876 (127.4.27.3:36321)
I20260812 14:53:34.558305 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.547032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:34997 (local address 127.4.27.2:36401)
0812 14:53:34.547102 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.547104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.549727 (+ 2623us) server_negotiation.cc:407] Connection header received
0812 14:53:34.549825 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.549826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.549839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.549867 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:53:34.550098 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.550803 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.557369 (+ 6566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.557905 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.557950 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.557961 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.558193 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.558236 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:53:34.559309 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.546841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36401 (local address 127.4.27.1:34997)
0812 14:53:34.549656 (+ 2815us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.549667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.549757 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.549931 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.549932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.549934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:53:34.550021 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.550023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.550924 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.550924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.557267 (+ 6343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.557270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.558017 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.558018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.558116 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.558120 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.558946 (+ 826us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.558996 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2792,"thread_start_us":92,"threads_started":1}
I20260812 14:53:34.559793 25132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1212b0feac664c5680ad42723f29c845" candidate_uuid: "17642f7fd20a4e0488e0911bff04ab75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" is_pre_election: true
I20260812 14:53:34.559959 25132 raft_consensus.cc:2454] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17642f7fd20a4e0488e0911bff04ab75 in term 0.
I20260812 14:53:34.560186 24771 leader_election.cc:302] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17642f7fd20a4e0488e0911bff04ab75, 5d5da0ea20f34d59bfdbe1e0b3bbd304; no voters:
I20260812 14:53:34.560317 25499 raft_consensus.cc:2781] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:53:34.560360 25499 raft_consensus.cc:482] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:53:34.560385 25499 raft_consensus.cc:3037] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:53:34.561945 25499 raft_consensus.cc:504] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.562238 25132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1212b0feac664c5680ad42723f29c845" candidate_uuid: "17642f7fd20a4e0488e0911bff04ab75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304"
I20260812 14:53:34.562319 25132 raft_consensus.cc:3037] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:53:34.563148 25132 raft_consensus.cc:2454] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17642f7fd20a4e0488e0911bff04ab75 in term 1.
I20260812 14:53:34.563319 24771 leader_election.cc:302] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17642f7fd20a4e0488e0911bff04ab75, 5d5da0ea20f34d59bfdbe1e0b3bbd304; no voters:
I20260812 14:53:34.563517 25499 leader_election.cc:288] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [CANDIDATE]: Term 1 election: Requested vote from peers 5d5da0ea20f34d59bfdbe1e0b3bbd304 (127.4.27.2:36401), ecdf92a4a20e4782b0995d259e491876 (127.4.27.3:36321)
I20260812 14:53:34.563563 25499 raft_consensus.cc:2781] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:53:34.563598 25499 raft_consensus.cc:686] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [term 1 LEADER]: Becoming Leader. State: Replica: 17642f7fd20a4e0488e0911bff04ab75, State: Running, Role: LEADER
I20260812 14:53:34.563680 25499 consensus_queue.cc:237] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } }
I20260812 14:53:34.564474 24604 catalog_manager.cc:5495] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17642f7fd20a4e0488e0911bff04ab75 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "17642f7fd20a4e0488e0911bff04ab75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17642f7fd20a4e0488e0911bff04ab75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 42071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20260812 14:53:34.569689 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.547449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:60881 (local address 127.4.27.3:36321)
0812 14:53:34.547487 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.547489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.552399 (+ 4910us) server_negotiation.cc:407] Connection header received
0812 14:53:34.552492 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.552493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.552503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.552530 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:53:34.552735 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.553406 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.568598 (+ 15192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.569187 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.569256 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.569268 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.569552 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.569615 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:53:34.569954 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.547282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:36321 (local address 127.4.27.1:60881)
0812 14:53:34.552336 (+ 5054us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.552347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:53:34.552426 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:53:34.552592 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:53:34.552593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:53:34.552595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:53:34.552671 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.552673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.555668 (+ 2995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.555669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.556781 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:53:34.556784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.569365 (+ 12581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.569366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:53:34.569458 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.569463 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:53:34.569814 (+ 351us) client_negotiation.cc:240] Negotiation successful
0812 14:53:34.569859 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5034,"thread_start_us":81,"threads_started":1}
I20260812 14:53:34.570400 25380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1212b0feac664c5680ad42723f29c845" candidate_uuid: "17642f7fd20a4e0488e0911bff04ab75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdf92a4a20e4782b0995d259e491876"
I20260812 14:53:34.570503 25380 raft_consensus.cc:3037] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:53:34.571416 25380 raft_consensus.cc:2454] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17642f7fd20a4e0488e0911bff04ab75 in term 1.
I20260812 14:53:34.570274 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1212b0feac664c5680ad42723f29c845" candidate_uuid: "17642f7fd20a4e0488e0911bff04ab75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdf92a4a20e4782b0995d259e491876" is_pre_election: true
I20260812 14:53:34.571750 25382 raft_consensus.cc:2362] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 17642f7fd20a4e0488e0911bff04ab75 in term 1. Re-sending same reply.
I20260812 14:53:34.576122 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.573114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.4.27.1:42071)
0812 14:53:34.573149 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:53:34.573150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:53:34.573156 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:53:34.573184 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:53:34.573185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:53:34.573201 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:53:34.573256 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:53:34.573397 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.574051 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.575098 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:53:34.575362 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:53:34.575411 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:53:34.575597 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:53:34.575724 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:53:34.575791 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:53:34.576026 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 14:53:34.576045 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:53:34.579018 25382 raft_consensus.cc:1261] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 1 FOLLOWER]: Refusing update from remote peer 17642f7fd20a4e0488e0911bff04ab75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:53:34.579018 25132 raft_consensus.cc:1261] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 1 FOLLOWER]: Refusing update from remote peer 17642f7fd20a4e0488e0911bff04ab75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:53:34.579344 25514 consensus_queue.cc:1035] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d5da0ea20f34d59bfdbe1e0b3bbd304" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:53:34.579438 25499 consensus_queue.cc:1035] T 1212b0feac664c5680ad42723f29c845 P 17642f7fd20a4e0488e0911bff04ab75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdf92a4a20e4782b0995d259e491876" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:53:34.583179 25518 mvcc.cc:205] Tried to move back new op lower bound from 7317694114113081344 to 7317694114053562368. Current Snapshot: MvccSnapshot[applied={T|T < 7317694114113081344}]
I20260812 14:53:34.585124 25520 mvcc.cc:205] Tried to move back new op lower bound from 7317694114113081344 to 7317694114053562368. Current Snapshot: MvccSnapshot[applied={T|T < 7317694114113081344}]
I20260812 14:53:34.586746 25519 mvcc.cc:205] Tried to move back new op lower bound from 7317694114113081344 to 7317694114053562368. Current Snapshot: MvccSnapshot[applied={T|T < 7317694114113081344}]
I20260812 14:53:34.587339 24604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:53:34.587436 24604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:53:34.588105 24604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e939e48162c4d11a1ef5aac8488eaef: Sending DeleteTablet for 3 replicas of tablet 1212b0feac664c5680ad42723f29c845
I20260812 14:53:34.588577 25104 tablet_service.cc:1514] Processing DeleteTablet for tablet 1212b0feac664c5680ad42723f29c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:53:34 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:45318
I20260812 14:53:34.588791 24841 tablet_service.cc:1514] Processing DeleteTablet for tablet 1212b0feac664c5680ad42723f29c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:53:34 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:56342
I20260812 14:53:34.588984 25324 tablet_service.cc:1514] Processing DeleteTablet for tablet 1212b0feac664c5680ad42723f29c845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:53:34 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:58716
I20260812 14:53:34.589485 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 24736
I20260812 14:53:34.603452 25534 tablet_replica.cc:330] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: stopping tablet replica
I20260812 14:53:34.603581 25534 raft_consensus.cc:2229] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:53:34.603701 25534 raft_consensus.cc:2258] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:53:34.603828 25538 tablet_replica.cc:330] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: stopping tablet replica
I20260812 14:53:34.603926 25538 raft_consensus.cc:2229] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:53:34.604019 25538 raft_consensus.cc:2258] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:53:34.605125 25538 ts_tablet_manager.cc:1905] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:53:34.606714 25538 ts_tablet_manager.cc:1918] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:53:34.606789 25538 log.cc:1196] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/1212b0feac664c5680ad42723f29c845
I20260812 14:53:34.607335 25538 ts_tablet_manager.cc:1939] T 1212b0feac664c5680ad42723f29c845 P ecdf92a4a20e4782b0995d259e491876: Deleting consensus metadata
I20260812 14:53:34.607756 24572 catalog_manager.cc:4841] TS ecdf92a4a20e4782b0995d259e491876 (127.4.27.3:36321): tablet 1212b0feac664c5680ad42723f29c845 (table test-table [id=30403fbd0a5144539f23a54b0ceed528]) successfully deleted
I20260812 14:53:34.608072 25534 ts_tablet_manager.cc:1905] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:53:34.615111 25534 ts_tablet_manager.cc:1918] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:53:34.617142 25534 log.cc:1196] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/1212b0feac664c5680ad42723f29c845
I20260812 14:53:34.617725 25534 ts_tablet_manager.cc:1939] T 1212b0feac664c5680ad42723f29c845 P 5d5da0ea20f34d59bfdbe1e0b3bbd304: Deleting consensus metadata
I20260812 14:53:34.618157 24576 catalog_manager.cc:4841] TS 5d5da0ea20f34d59bfdbe1e0b3bbd304 (127.4.27.2:36401): tablet 1212b0feac664c5680ad42723f29c845 (table test-table [id=30403fbd0a5144539f23a54b0ceed528]) successfully deleted
W20260812 14:53:34.620817 24572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:42071 (error 108)
I20260812 14:53:34.621477 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 24971
I20260812 14:53:34.715152 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:53:34.634696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:42071 (local address 127.0.0.1:56372)
0812 14:53:34.715023 (+ 80327us) negotiation.cc:107] Waiting for socket to connect
0812 14:53:34.715099 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:42071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80284}
W20260812 14:53:34.715844 24572 catalog_manager.cc:4584] TS 17642f7fd20a4e0488e0911bff04ab75 (127.4.27.1:42071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1212b0feac664c5680ad42723f29c845: Network error: Client connection negotiation failed: client connection to 127.4.27.1:42071: connect: Connection refused (error 111)
I20260812 14:53:34.726956 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 25222
I20260812 14:53:34.740123 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 24550
I20260812 14:53:34.754905 4204 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 14:53:35.100024 4204 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 14:53:35.100111 4204 mini_ranger.cc:66] Stopping Ranger...
I20260812 14:53:35.443704 4204 mini_ranger.cc:68] Stopped Ranger
2026-08-12 14:53:35.443 UTC [23769] LOG: received smart shutdown request
2026-08-12 14:53:35.447 UTC [23769] LOG: background worker "logical replication launcher" (PID 23789) exited with exit code 1
2026-08-12 14:53:35.447 UTC [23778] LOG: shutting down
2026-08-12 14:53:35.572 UTC [23769] LOG: database system is shut down
2026-08-12T14:53:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44069 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:53:35 dist-test-slave-6zbq krb5kdc[25559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:53:35 dist-test-slave-6zbq krb5kdc[25559](info): set up 2 sockets
Aug 12 14:53:35 dist-test-slave-6zbq krb5kdc[25559](info): commencing operation
krb5kdc: starting...
W20260812 14:53:36.843844 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:53:36 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:53:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/rangeradmin_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/rangeradmin_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/rangerlookup_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/rangerlookup_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
I20260812 14:53:36.986071 4204 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktWKW5U/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/postgres -l logfile start
2026-08-12 14:53:39.073 UTC [25613] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 14:53:39.073 UTC [25613] LOG: listening on IPv4 address "127.4.27.20", port 58371
2026-08-12 14:53:39.076 UTC [25613] LOG: listening on Unix socket "/tmp/.s.PGSQL.58371"
2026-08-12 14:53:39.102 UTC [25617] LOG: database system was shut down at 2026-08-12 14:53:37 UTC
2026-08-12 14:53:39.105 UTC [25613] LOG: database system is ready to accept connections
I20260812 14:53:40.203604 4204 mini_postgres.cc:96] Postgres bound to 58371
2026-08-12 14:53:40.210 UTC [25637] FATAL: database "slave" does not exist
127.4.27.20:58371 - accepting connections
I20260812 14:53:40.210644 4204 mini_ranger.cc:161] Starting Ranger...
I20260812 14:53:40.224429 4204 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 14:53:40.595733 4204 mini_ranger.cc:87] Created ranger Postgres database
I20260812 14:53:40.595810 4204 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-admin
2026-08-12 14:53:40,861 [I] DB FLAVOR :POSTGRES
2026-08-12 14:53:40,861 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:53:40,861 [I] Checking connection..
2026-08-12 14:53:40,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 14:53:41,098 [I] Checking connection passed.
2026-08-12 14:53:41,098 [I] --------- Verifying version history table ---------
2026-08-12 14:53:41,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:53:41,347 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:53:41,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:53:41,592 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 14:53:41,593 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 14:53:41,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 14:53:41.836 UTC [25744] WARNING: there is no transaction in progress
2026-08-12 14:53:41,844 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 14:53:41,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 14:53:42,094 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 14:53:42,094 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:53:42,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 14:53:42,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 14:53:42,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6zbq.c.gcp-upstream.internal','N') ;"
2026-08-12 14:53:42,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:53:43,064 [I] Table x_portal_user does not exist in database ranger
2026-08-12 14:53:43,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:53:43,302 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 14:53:43,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:53:43,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:53:43,788 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 14:53:43,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 14:53:44.657 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.670 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.684 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.696 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.709 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.724 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.770 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.777 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.789 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.798 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.807 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.819 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.827 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.831 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.836 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:44.842 UTC [25959] WARNING: there is no transaction in progress
2026-08-12 14:53:45,359 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 14:53:45,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 14:53:45,601 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 14:53:45,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 14:53:45,893 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 14:53:45,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 14:53:46,155 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 14:53:46,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 14:53:46,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 14:53:46,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6zbq.c.gcp-upstream.internal';"
2026-08-12 14:53:46,887 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 14:53:46,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 14:53:47,138 [I] DB_PATCHES have already been applied
I20260812 14:53:47.143713 4204 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktWKW5U/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 14:53:47.143772 4204 mini_ranger.cc:193] Using host: 127.4.27.20
I20260812 14:53:47.147410 4204 mini_ranger.cc:227] Ranger admin URL: http://127.4.27.20:47731
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:53:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 14:53:48.444384 4204 mini_ranger.cc:160] Time spent starting Ranger: real 8.234s user 0.001s sys 0.008s
Aug 12, 2026 2:54:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:54:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:54:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:54:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:54:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 14:54:09.031997 4204 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.27.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.27.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/rangerkms_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.27.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/rangerkms_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.27.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.27.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.27.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/HTTP_127.4.27.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 14:54:09 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546449, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 14:54:09.101703 4204 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 14:54:09.110162 4204 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 14:54:09.749449 4204 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 14:54:09.749531 4204 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms
I20260812 14:54:09.749547 4204 mini_ranger_kms.cc:227] Using postgres at 127.4.27.20:58371
2026-08-12 14:54:09,909 [I] DB FLAVOR :POSTGRES
2026-08-12 14:54:09,909 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 14:54:09,909 [I] Checking connection
2026-08-12 14:54:09,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 14:54:10,157 [I] connection success
2026-08-12 14:54:10,157 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 14:54:10,157 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 14:54:10,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 14:54:10,438 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 14:54:10,438 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 14:54:10,438 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 14:54:10,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.27.20:58371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 14:54:10,684 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 14:54:11.104548 4204 mini_ranger_kms.cc:310] Created kms service
I20260812 14:54:11.280134 4204 mini_ranger_kms.cc:326] Created kudu user
I20260812 14:54:11.317114 4204 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 14:54:11 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546449, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:11.688117 4204 mini_ranger_kms.cc:384] Added ranger policy
I20260812 14:54:11.688253 4204 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktWKW5U/build/release/bin/postgresql.jar:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktWKW5U/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 14:54:11.688277 4204 mini_ranger_kms.cc:240] Using host: 127.4.27.20
I20260812 14:54:11.692426 4204 mini_ranger_kms.cc:276] Ranger KMS PID: 26290
I20260812 14:54:11.692499 4204 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.27.20:45921
Aug 12, 2026 2:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:54:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:54:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 14:54:13 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786546453, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.27.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:14.345415 4204 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.244s user 0.002s sys 0.141s
I20260812 14:54:14.345500 4204 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 14:54:14.345546 4204 mini_ranger_kms.cc:401] 127.4.27.20:45921/kms/v1/keys
Aug 12 14:54:14 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546453, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.27.20@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
Aug 12 14:54:14 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546453, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.27.20@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
Aug 12 14:54:17 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:54:17.120935 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:33219
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:33219
--ranger_config_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:17.224809 26389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:17.225008 26389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:17.225028 26389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:17.226248 26389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:17.226290 26389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:17.226305 26389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:17.226315 26389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:17.226325 26389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:17.227905 26389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.27.20:45921/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:33219
--ranger_config_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:17.228123 26389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:17.228336 26389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:17.233321 26400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:17.233282 26398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:17.233331 26389 server_base.cc:1034] running on GCE node
W20260812 14:54:17.233297 26403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:17.233676 26389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:17.233992 26389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:17.235148 26389 hybrid_clock.cc:648] HybridClock initialized: now 1786546457235134 us; error 28 us; skew 500 ppm
Aug 12 14:54:17 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:17.238153 26389 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:17.238591 26389 webserver.cc:460] Webserver started at http://127.4.27.62:41729/ using document root <none> and password file <none>
I20260812 14:54:17.238782 26389 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:17.238822 26389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:17.238938 26389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:54:17 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546457, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:17.283097 26389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "65a22e608f924788896970311ef00900"
format_stamp: "Formatted at 2026-08-12 14:54:17 on dist-test-slave-6zbq"
server_key: "b30f785f338ec4d7a7e1b13f76a386b8"
server_key_iv: "908b4d18d636d904b2ae815a3baeddd6"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:17.283461 26389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "65a22e608f924788896970311ef00900"
format_stamp: "Formatted at 2026-08-12 14:54:17 on dist-test-slave-6zbq"
server_key: "b30f785f338ec4d7a7e1b13f76a386b8"
server_key_iv: "908b4d18d636d904b2ae815a3baeddd6"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:17.284897 26389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:17.298691 26415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:17.298863 26389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:17.298943 26389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "65a22e608f924788896970311ef00900"
format_stamp: "Formatted at 2026-08-12 14:54:17 on dist-test-slave-6zbq"
server_key: "b30f785f338ec4d7a7e1b13f76a386b8"
server_key_iv: "908b4d18d636d904b2ae815a3baeddd6"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:17.299008 26389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:17.311909 26389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:17.313572 26389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:17.313716 26389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:17.318401 26389 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:33219
I20260812 14:54:17.318461 26508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:33219 every 8 connection(s)
I20260812 14:54:17.318784 26389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:17.319499 26509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:17.322625 26509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Bootstrap starting.
I20260812 14:54:17.323328 26509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:17.323644 26509 log.cc:824] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:17.324384 26509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: No bootstrap required, opened a new log
I20260812 14:54:17.326264 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 26389
I20260812 14:54:17.326368 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:54:17.326545 4204 external_mini_cluster.cc:1428] Setting key 9925527519a4eefd8dcb9b155c89ac92
I20260812 14:54:17.327231 26509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:17.327416 26509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:17.327453 26509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a22e608f924788896970311ef00900, State: Initialized, Role: FOLLOWER
I20260812 14:54:17.327642 26509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:17.327739 26509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:17.327801 26509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:17.327878 26509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:17.328526 26509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:17.328637 26509 leader_election.cc:302] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a22e608f924788896970311ef00900; no voters:
I20260812 14:54:17.328846 26509 leader_election.cc:288] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:54:17.328892 26517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:17.329109 26517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 LEADER]: Becoming Leader. State: Replica: 65a22e608f924788896970311ef00900, State: Running, Role: LEADER
I20260812 14:54:17.329195 26509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:17.329231 26517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:17.330504 26519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a22e608f924788896970311ef00900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } } }
I20260812 14:54:17.330636 26519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:17.331368 26509 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:54:17.331569 26518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a22e608f924788896970311ef00900. Latest consensus state: current_term: 1 leader_uuid: "65a22e608f924788896970311ef00900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } } }
I20260812 14:54:17.331629 26518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: This master's current role is: LEADER
Aug 12 14:54:17 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:17.335124 26530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:17.337062 26530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:17.338493 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:17.326985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.4.27.62:33219)
0812 14:54:17.327230 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:17.327237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:17.327261 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:54:17.327363 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:17.327368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:17.327419 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:17.327544 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:17.328255 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:17.329420 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:17.330498 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:17.331102 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:17.331206 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:17.334356 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:17.334400 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:17.334400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:17.334468 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:17.336383 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:17.336788 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:17.336792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:17.336792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:17.336897 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:17.337124 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:17.337125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:17.337125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:17.337509 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:17.337676 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:17.338089 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 14:54:17.338244 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":118,"threads_started":1}
I20260812 14:54:17.342094 26530 catalog_manager.cc:1347] Generated new cluster ID: 964c00eb964c415cb0e75485b2d9bf99
I20260812 14:54:17.342145 26530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:54:17.358563 26530 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:54:17.359333 26530 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:17.364034 26530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Generated new TSK 0
I20260812 14:54:17.364252 26530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:54:18.070370 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:33219
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:18.168154 26582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:18.168334 26582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:18.168355 26582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:18.169503 26582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:18.169559 26582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:18.169597 26582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:54:18.171352 26582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.27.20:45921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:18.171569 26582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:18.171844 26582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:18.176741 26596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:18.176822 26599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:18.176741 26597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:18.177219 26582 server_base.cc:1034] running on GCE node
I20260812 14:54:18.177508 26582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:18.177772 26582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:18.178949 26582 hybrid_clock.cc:648] HybridClock initialized: now 1786546458178923 us; error 42 us; skew 500 ppm
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:18.181746 26582 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:54:18.182168 26582 webserver.cc:460] Webserver started at http://127.4.27.1:46669/ using document root <none> and password file <none>
I20260812 14:54:18.182430 26582 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:18.182556 26582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:18.182735 26582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:18.196028 26582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "46a4d5c5a09449ff96f32a0b33c601b3"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "dd210ff271645d9871fc4d3164348050"
server_key_iv: "d1ddaf47d24e8b96afa6715e4cfa868d"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.196471 26582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "46a4d5c5a09449ff96f32a0b33c601b3"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "dd210ff271645d9871fc4d3164348050"
server_key_iv: "d1ddaf47d24e8b96afa6715e4cfa868d"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.202099 26582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 14:54:18.225750 26614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.226020 26582 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260812 14:54:18.226184 26582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "46a4d5c5a09449ff96f32a0b33c601b3"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "dd210ff271645d9871fc4d3164348050"
server_key_iv: "d1ddaf47d24e8b96afa6715e4cfa868d"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.226402 26582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:18.264176 26582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:18.266515 26582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:18.266646 26582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:18.266870 26582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:18.267218 26582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:18.267253 26582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.267282 26582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:18.267297 26582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.273451 26582 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43453
I20260812 14:54:18.273818 26582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:54:18.276336 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 26582
I20260812 14:54:18.276430 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:54:18.276633 4204 external_mini_cluster.cc:1428] Setting key f70b25d85b4e77b25bd6671b4e1eaa7a
I20260812 14:54:18.279834 26819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43453 every 8 connection(s)
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
I20260812 14:54:18.293385 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:18.280467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:35959 (local address 127.4.27.62:33219)
0812 14:54:18.283052 (+ 2585us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:18.283054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:18.283062 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:18.283086 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:18.283087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:18.283099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:18.283183 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:18.283951 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.284747 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.286404 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.286716 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.286776 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:18.289984 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:18.289985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:18.289986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:18.289994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:18.291742 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:18.292241 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:18.292242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:18.292243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:18.292315 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:18.292595 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:18.292596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:18.292596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:18.292725 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:18.292786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:18.293199 (+ 413us) server_negotiation.cc:299] Negotiation successful
0812 14:54:18.293289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2554,"thread_start_us":97,"threads_started":1}
I20260812 14:54:18.293608 26828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:18.280427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:33219 (local address 127.4.27.1:35959)
0812 14:54:18.282189 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:18.282238 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:18.282869 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:18.283323 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:18.283327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:18.283568 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:18.283844 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:18.283851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.284847 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.284849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:18.286072 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:18.286075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.286856 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.286857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:18.286999 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:18.287520 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:18.287530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:18.289869 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:18.291817 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:18.291825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:18.291827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:18.292157 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:18.292392 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:18.292392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:18.292393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:18.292531 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:18.292791 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:18.292797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:18.292976 (+ 179us) client_negotiation.cc:769] Sending connection context
0812 14:54:18.293131 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 14:54:18.293327 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
I20260812 14:54:18.296792 26821 heartbeater.cc:347] Connected to a master server at 127.4.27.62:33219
I20260812 14:54:18.296898 26821 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:18.297091 26821 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:18.297679 26446 ts_manager.cc:194] Registered new tserver with Master: 46a4d5c5a09449ff96f32a0b33c601b3 (127.4.27.1:43453)
I20260812 14:54:18.298740 26446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:35959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:54:18.326728 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:33219
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 14:54:18.451711 26835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:18.451987 26835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:18.452020 26835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:18.453712 26835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:18.453759 26835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:18.453794 26835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:54:18.455956 26835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.27.20:45921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:18.456200 26835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:18.456532 26835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:18.462918 26847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:18.463044 26845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:18.463236 26835 server_base.cc:1034] running on GCE node
W20260812 14:54:18.463409 26844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:18.463709 26835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:18.464001 26835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:18.465176 26835 hybrid_clock.cc:648] HybridClock initialized: now 1786546458465155 us; error 34 us; skew 500 ppm
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:18.468211 26835 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:54:18.468770 26835 webserver.cc:460] Webserver started at http://127.4.27.2:40221/ using document root <none> and password file <none>
I20260812 14:54:18.469058 26835 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:18.469128 26835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:18.469280 26835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:18.482255 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "c1d6677e2d044d2e93d57e20461d35b5"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "0cc601e7f13d048079cef550d47be5fe"
server_key_iv: "e38e668d54c2b8bc0d1ee400b680d1f5"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.482805 26835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "c1d6677e2d044d2e93d57e20461d35b5"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "0cc601e7f13d048079cef550d47be5fe"
server_key_iv: "e38e668d54c2b8bc0d1ee400b680d1f5"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.484759 26835 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:18.503316 26870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.503552 26835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:18.503643 26835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "c1d6677e2d044d2e93d57e20461d35b5"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "0cc601e7f13d048079cef550d47be5fe"
server_key_iv: "e38e668d54c2b8bc0d1ee400b680d1f5"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.503726 26835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:18.516605 26835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:18.518870 26835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:18.519016 26835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:18.519244 26835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:18.519611 26835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:18.519657 26835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.519686 26835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:18.519750 26835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.526908 26835 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36629
I20260812 14:54:18.527344 26835 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:54:18.528585 27070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36629 every 8 connection(s)
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:18.535306 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 26835
I20260812 14:54:18.535399 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:54:18.535609 4204 external_mini_cluster.cc:1428] Setting key 26ec2bcddb172eaa53e4df7afe51cfd4
I20260812 14:54:18.542148 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:18.528416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:43863 (local address 127.4.27.62:33219)
0812 14:54:18.528464 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:18.528466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:18.528477 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:54:18.529097 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:18.529099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:18.529112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:18.529212 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:18.529937 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.530832 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.532352 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.532674 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.532752 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:18.538927 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:18.538929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:18.538929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:18.538936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:18.540522 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:18.541017 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:18.541019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:18.541019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:18.541080 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:18.541396 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:18.541397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:18.541397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:18.541533 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:18.541576 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:18.541972 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 14:54:18.542062 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:54:18.542923 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:18.528098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:33219 (local address 127.4.27.2:43863)
0812 14:54:18.528304 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:18.528351 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:18.528975 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:18.529295 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:18.529300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:18.529537 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:18.529809 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:18.529815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.530935 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.530936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:18.532243 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:18.532246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.532787 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.532789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:18.532967 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:18.533516 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:18.533526 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:18.538817 (+ 5291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:18.540619 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:18.540627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:18.540629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:18.540921 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:18.541179 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:18.541180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:18.541180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:18.541316 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:18.541622 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:18.541628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:18.541817 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 14:54:18.542489 (+ 672us) client_negotiation.cc:240] Negotiation successful
0812 14:54:18.542698 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260812 14:54:18.543561 27071 heartbeater.cc:347] Connected to a master server at 127.4.27.62:33219
I20260812 14:54:18.543664 27071 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:18.543840 27071 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:18.544214 26446 ts_manager.cc:194] Registered new tserver with Master: c1d6677e2d044d2e93d57e20461d35b5 (127.4.27.2:36629)
I20260812 14:54:18.544652 26446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:43863
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:54:18.569554 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:33219
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:18.694415 27078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:18.694777 27078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:18.694877 27078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:18.696688 27078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:18.696803 27078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:18.696889 27078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:54:18.699291 27078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.27.20:45921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:18.699755 27078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:18.700078 27078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:18.706463 27087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:18.706713 27088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:18.706827 27078 server_base.cc:1034] running on GCE node
W20260812 14:54:18.707028 27092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:18.707280 27078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:18.707567 27078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:18.708776 27078 hybrid_clock.cc:648] HybridClock initialized: now 1786546458708754 us; error 35 us; skew 500 ppm
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:18.712285 27078 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:54:18.713076 27078 webserver.cc:460] Webserver started at http://127.4.27.3:41363/ using document root <none> and password file <none>
I20260812 14:54:18.713361 27078 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:18.713435 27078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:18.713629 27078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:18.734227 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "09e7cfc8275948c39049cdebdc2a6323"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "fa5cae94d4b373c155e38e1820aad99d"
server_key_iv: "19dcb7b3d2682cc4919817494cdafe30"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.735297 27078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "09e7cfc8275948c39049cdebdc2a6323"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "fa5cae94d4b373c155e38e1820aad99d"
server_key_iv: "19dcb7b3d2682cc4919817494cdafe30"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.739337 27078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260812 14:54:18.758983 27103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.759569 27078 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260812 14:54:18.759713 27078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "09e7cfc8275948c39049cdebdc2a6323"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "fa5cae94d4b373c155e38e1820aad99d"
server_key_iv: "19dcb7b3d2682cc4919817494cdafe30"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:18.759807 27078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:18.798177 27078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:18.800418 27078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:18.800588 27078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:18.800855 27078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:18.801545 27078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:18.801613 27078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:18.801648 27078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:18.801724 27078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 14:54:18.809831 27078 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42249
I20260812 14:54:18.810101 27317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42249 every 8 connection(s)
I20260812 14:54:18.810389 27078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:54:18.813763 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 27078
I20260812 14:54:18.813864 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:54:18.814070 4204 external_mini_cluster.cc:1428] Setting key d07684befe9959eb7fc9a4320a80f3b7
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:18.821826 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:18.811689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:33751 (local address 127.4.27.62:33219)
0812 14:54:18.811738 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:18.811740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:18.812016 (+ 276us) server_negotiation.cc:407] Connection header received
0812 14:54:18.812693 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:18.812694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:18.812709 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:18.812793 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:18.813598 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.814382 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.815848 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.816148 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.816194 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:18.818775 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:18.818776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:18.818777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:18.818784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:18.820355 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:18.820822 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:18.820823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:18.820824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:18.820877 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:18.821135 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:18.821136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:18.821137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:18.821260 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:18.821299 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:18.821654 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 14:54:18.821740 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:54:18.822383 27326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:18.811686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:33219 (local address 127.4.27.3:33751)
0812 14:54:18.811857 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:18.811907 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:18.812557 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:18.812933 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:18.812938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:18.813180 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:18.813460 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:18.813466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.814498 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.814499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:18.815746 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:18.815749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:18.816300 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:18.816301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:18.816438 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:18.817036 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:18.817046 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:18.818657 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:18.820452 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:18.820460 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:18.820462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:18.820739 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:18.820938 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:18.820939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:18.820939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:18.821052 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:18.821342 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:18.821348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:18.821518 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 14:54:18.821908 (+ 390us) client_negotiation.cc:240] Negotiation successful
0812 14:54:18.822126 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260812 14:54:18.823814 27318 heartbeater.cc:347] Connected to a master server at 127.4.27.62:33219
I20260812 14:54:18.823913 27318 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:18.824097 27318 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:18.825862 26446 ts_manager.cc:194] Registered new tserver with Master: 09e7cfc8275948c39049cdebdc2a6323 (127.4.27.3:42249)
I20260812 14:54:18.826407 26446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:33751
I20260812 14:54:18.827291 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:54:18.827989 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 26582
Aug 12 14:54:18 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:18.874406 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 26835
I20260812 14:54:18.882182 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 27078
I20260812 14:54:18.912423 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 26389
I20260812 14:54:18.931452 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:33219
--webserver_interface=127.4.27.62
--webserver_port=41729
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:33219
--ranger_config_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:19.020515 27328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:19.020730 27328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:19.020763 27328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:19.023873 27328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:19.023974 27328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:19.024001 27328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:19.024024 27328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:19.024040 27328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:19.026521 27328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:33219
--ranger_config_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:33219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=41729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:19.026791 27328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:19.027027 27328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:19.032686 27338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:19.032680 27339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:19.032680 27342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:19.033070 27328 server_base.cc:1034] running on GCE node
I20260812 14:54:19.033339 27328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:19.033619 27328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:19.034783 27328 hybrid_clock.cc:648] HybridClock initialized: now 1786546459034767 us; error 31 us; skew 500 ppm
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:19.037338 27328 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:19.037864 27328 webserver.cc:460] Webserver started at http://127.4.27.62:41729/ using document root <none> and password file <none>
I20260812 14:54:19.038123 27328 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:19.038194 27328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:19.113719 27328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:19.114426 27353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.114588 27328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:19.114683 27328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "65a22e608f924788896970311ef00900"
format_stamp: "Formatted at 2026-08-12 14:54:17 on dist-test-slave-6zbq"
server_key: "b30f785f338ec4d7a7e1b13f76a386b8"
server_key_iv: "908b4d18d636d904b2ae815a3baeddd6"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:19.115098 27328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:19.129205 27328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:19.130728 27328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:19.130846 27328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:19.135156 27328 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:33219
I20260812 14:54:19.135309 27444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:33219 every 8 connection(s)
I20260812 14:54:19.135612 27328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:19.136925 27445 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:54:19.137527 27445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Bootstrap starting.
I20260812 14:54:19.140573 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 27328
I20260812 14:54:19.140910 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:43453
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:33219
--webserver_port=46669
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:19.147179 27445 log.cc:824] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:19.148821 27445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:54:19.149106 27445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Bootstrap complete.
I20260812 14:54:19.151723 27445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:19.151961 27445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a22e608f924788896970311ef00900, State: Initialized, Role: FOLLOWER
I20260812 14:54:19.152103 27445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:19.152173 27445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:19.152216 27445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:19.152261 27445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:54:19.153230 27445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:19.153318 27445 leader_election.cc:302] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a22e608f924788896970311ef00900; no voters:
I20260812 14:54:19.153473 27445 leader_election.cc:288] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:54:19.153535 27449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:54:19.153731 27445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:19.153743 27449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [term 2 LEADER]: Becoming Leader. State: Replica: 65a22e608f924788896970311ef00900, State: Running, Role: LEADER
I20260812 14:54:19.153965 27449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } }
I20260812 14:54:19.154250 27450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65a22e608f924788896970311ef00900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } } }
I20260812 14:54:19.154316 27450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:19.154548 27451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a22e608f924788896970311ef00900. Latest consensus state: current_term: 2 leader_uuid: "65a22e608f924788896970311ef00900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a22e608f924788896970311ef00900" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33219 } } }
I20260812 14:54:19.154657 27451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:19.154657 27469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:19.155309 27469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:19.155485 27469 catalog_manager.cc:1259] Loaded cluster ID: 964c00eb964c415cb0e75485b2d9bf99
I20260812 14:54:19.155566 27469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:54:19.155622 27445 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 14:54:19.159058 27469 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:19.159670 27469 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Loaded TSK: 0
I20260812 14:54:19.160036 27469 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:54:19.254982 27447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:19.255246 27447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:19.255285 27447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:19.256877 27447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:19.256982 27447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:19.257061 27447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:54:19.259449 27447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:43453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=46669
--tserver_master_addrs=127.4.27.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:19.259944 27447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:19.260277 27447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:19.267869 27525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:19.268649 27447 server_base.cc:1034] running on GCE node
W20260812 14:54:19.270649 27522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:19.270704 27520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:19.271049 27447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:19.271328 27447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:19.278687 27447 hybrid_clock.cc:648] HybridClock initialized: now 1786546459278651 us; error 45 us; skew 500 ppm
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:19.281910 27447 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:54:19.282473 27447 webserver.cc:460] Webserver started at http://127.4.27.1:46669/ using document root <none> and password file <none>
I20260812 14:54:19.282811 27447 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:19.282903 27447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:19.304754 27447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:19.306155 27543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.306378 27447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:19.306480 27447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "46a4d5c5a09449ff96f32a0b33c601b3"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "dd210ff271645d9871fc4d3164348050"
server_key_iv: "d1ddaf47d24e8b96afa6715e4cfa868d"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:19.306934 27447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:19.326495 27447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:19.328879 27447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:19.329080 27447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:19.329407 27447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:19.329833 27447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:19.329928 27447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.329995 27447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:19.330049 27447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.338790 27447 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43453
I20260812 14:54:19.338994 27762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43453 every 8 connection(s)
I20260812 14:54:19.339318 27447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:54:19.342785 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 27447
I20260812 14:54:19.343075 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:36629
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:33219
--webserver_port=40221
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:19.352363 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.340016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:33219 (local address 127.4.27.1:37917)
0812 14:54:19.340191 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:19.340240 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:19.340840 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:19.341391 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:19.341396 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:19.341622 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:19.341895 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:19.341902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.343486 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.343487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:19.344579 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:19.344583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.345513 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.345515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:19.345660 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.346243 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:19.346254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:19.348252 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:19.350465 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:19.350473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:19.350475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:19.350759 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:19.351049 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:19.351050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:19.351050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:19.351199 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:19.351694 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:19.351701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:19.351852 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 14:54:19.351962 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 14:54:19.352132 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260812 14:54:19.352893 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.340171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:37917 (local address 127.4.27.62:33219)
0812 14:54:19.340365 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:19.340374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:19.340410 (+ 36us) server_negotiation.cc:407] Connection header received
0812 14:54:19.341101 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:19.341109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:19.341176 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:19.341325 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:19.342031 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.343366 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.344677 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.345292 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.345393 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.348379 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:19.348383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:19.348384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:19.348397 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:19.350373 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.350857 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.350861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.350861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.350971 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.351281 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.351282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.351282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.351580 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:19.351663 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:19.352567 (+ 904us) server_negotiation.cc:299] Negotiation successful
0812 14:54:19.352699 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260812 14:54:19.353631 27763 heartbeater.cc:347] Connected to a master server at 127.4.27.62:33219
I20260812 14:54:19.353737 27763 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.353929 27763 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
W20260812 14:54:19.354943 27763 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:33219 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:54:19.355026 27763 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.355099 27763 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:19.355484 27763 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.355569 27763 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
W20260812 14:54:19.355912 27763 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:54:19.454300 27768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:19.454538 27768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:19.454560 27768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:19.455729 27768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:19.455776 27768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:19.455801 27768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:54:19.457261 27768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:36629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=40221
--tserver_master_addrs=127.4.27.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:19.457458 27768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:19.457659 27768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:19.462807 27779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:19.462819 27782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:19.464066 27768 server_base.cc:1034] running on GCE node
W20260812 14:54:19.466466 27780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:19.466686 27768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:19.466965 27768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:19.468149 27768 hybrid_clock.cc:648] HybridClock initialized: now 1786546459468119 us; error 48 us; skew 500 ppm
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:19.475692 27768 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:54:19.476262 27768 webserver.cc:460] Webserver started at http://127.4.27.2:40221/ using document root <none> and password file <none>
I20260812 14:54:19.476549 27768 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:19.476635 27768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:19.495894 27768 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260812 14:54:19.496688 27795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.496881 27768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:19.496968 27768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "c1d6677e2d044d2e93d57e20461d35b5"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "0cc601e7f13d048079cef550d47be5fe"
server_key_iv: "e38e668d54c2b8bc0d1ee400b680d1f5"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:19.498136 27768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:19.536876 27768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:19.539332 27768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:19.539499 27768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:19.539762 27768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:19.540184 27768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:19.540264 27768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.540323 27768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:19.540366 27768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.547113 27768 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36629
I20260812 14:54:19.547284 28000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36629 every 8 connection(s)
I20260812 14:54:19.547581 27768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:54:19.549702 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 27768
I20260812 14:54:19.550004 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.27.20:45921/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:42249
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:33219
--webserver_port=41363
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:19.569986 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.548335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:46893 (local address 127.4.27.62:33219)
0812 14:54:19.548386 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:19.548389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:19.548609 (+ 220us) server_negotiation.cc:407] Connection header received
0812 14:54:19.549480 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:19.549481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:19.549498 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:19.549528 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:19.551499 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.552264 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.553909 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.554232 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.554320 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.566822 (+ 12502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:19.566824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:19.566824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:19.566833 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:19.568462 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.568933 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.568934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.568935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.568988 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.569252 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.569253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.569253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.569387 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:19.569427 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:19.569799 (+ 372us) server_negotiation.cc:299] Negotiation successful
0812 14:54:19.569891 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:54:19.570132 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.548318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:33219 (local address 127.4.27.2:46893)
0812 14:54:19.548473 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:19.548514 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:19.549345 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:19.549719 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:19.549724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:19.551091 (+ 1367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:19.551381 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:19.551388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.552455 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.552456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:19.553807 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:19.553811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.554468 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.554469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:19.554901 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.560516 (+ 5615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:19.560527 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:19.566687 (+ 6160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:19.568553 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:19.568561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:19.568564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:19.568850 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:19.569051 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:19.569052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:19.569052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:19.569172 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:19.569468 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:19.569473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:19.569635 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 14:54:19.569744 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 14:54:19.569899 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260812 14:54:19.570731 28001 heartbeater.cc:347] Connected to a master server at 127.4.27.62:33219
I20260812 14:54:19.570849 28001 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.571056 28001 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
W20260812 14:54:19.571748 28001 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:33219 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:54:19.571831 28001 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.571903 28001 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:19.572216 28001 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.572321 28001 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
W20260812 14:54:19.572597 28001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:54:19.658752 28006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:19.658972 28006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:19.659004 28006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:19.660876 28006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:19.660938 28006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:19.660979 28006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:54:19.663066 28006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.27.20:45921/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=41363
--tserver_master_addrs=127.4.27.62:33219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:19.663318 28006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:19.663537 28006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:19.668285 28024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:19.668478 28028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:19.668274 28022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:19.669026 28006 server_base.cc:1034] running on GCE node
I20260812 14:54:19.669284 28006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:19.669534 28006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:19.670738 28006 hybrid_clock.cc:648] HybridClock initialized: now 1786546459670723 us; error 26 us; skew 500 ppm
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:19.673382 28006 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:54:19.673823 28006 webserver.cc:460] Webserver started at http://127.4.27.3:41363/ using document root <none> and password file <none>
I20260812 14:54:19.674043 28006 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:19.674104 28006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:19.688699 28006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:19.689652 28038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.689844 28006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:19.689925 28006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "09e7cfc8275948c39049cdebdc2a6323"
format_stamp: "Formatted at 2026-08-12 14:54:18 on dist-test-slave-6zbq"
server_key: "fa5cae94d4b373c155e38e1820aad99d"
server_key_iv: "19dcb7b3d2682cc4919817494cdafe30"
server_key_version: "kuduclusterkey@0"
I20260812 14:54:19.690434 28006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:19.716715 28006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:19.719185 28006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:19.719390 28006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:19.719661 28006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:19.720094 28006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:19.720274 28006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.720402 28006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:19.720464 28006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:19.727563 28006 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42249
I20260812 14:54:19.727932 28006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:54:19.727959 28258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42249 every 8 connection(s)
Aug 12 14:54:19 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:19.735459 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 28006
I20260812 14:54:19.740968 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.728788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:42405 (local address 127.4.27.62:33219)
0812 14:54:19.728858 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:19.728860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:19.729056 (+ 196us) server_negotiation.cc:407] Connection header received
0812 14:54:19.729770 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:19.729771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:19.729783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:19.729839 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:19.730529 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.731004 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.732450 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.732622 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.732676 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.734657 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:19.734659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:19.734659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:19.734667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:19.739351 (+ 4684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.739858 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.739859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.739860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.739923 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.740207 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.740208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.740209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.740351 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:19.740395 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:19.740777 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 14:54:19.740880 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:54:19.741261 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.728739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:33219 (local address 127.4.27.3:42405)
0812 14:54:19.728914 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:19.728965 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:19.729630 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:19.729914 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:19.729918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:19.730139 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:19.730435 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:19.730442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.731096 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.731097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:19.732376 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:19.732379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.732726 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.732727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:19.732850 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.733350 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:19.733358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:19.734562 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:19.739427 (+ 4865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:19.739435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:19.739437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:19.739773 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:19.739998 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:19.739999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:19.739999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:19.740128 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:19.740445 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:19.740452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:19.740622 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 14:54:19.740881 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 14:54:19.741028 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260812 14:54:19.741806 28259 heartbeater.cc:347] Connected to a master server at 127.4.27.62:33219
I20260812 14:54:19.741918 28259 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.742110 28259 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
W20260812 14:54:19.742767 28259 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:33219 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 14:54:19.742841 28259 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.742901 28259 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:19.743201 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:19.735829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.4.27.62:33219)
0812 14:54:19.735945 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:19.735947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:19.736062 (+ 115us) server_negotiation.cc:407] Connection header received
0812 14:54:19.736116 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:19.736117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:19.736129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:19.736188 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:19.736709 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.737186 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.737879 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:19.738074 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:19.738126 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:19.739430 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:19.739431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:19.739432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:19.739439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:19.741717 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.742052 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.742054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.742054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.742104 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:19.742264 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:19.742264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:19.742265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:19.742421 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:19.742573 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:19.743032 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 14:54:19.743112 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260812 14:54:19.743263 28259 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:19.743330 28259 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
W20260812 14:54:19.743701 28259 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 14:54:20 dist-test-slave-6zbq krb5kdc[25559](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:20.356093 27763 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:20.356252 27763 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:20.356645 27379 ts_manager.cc:194] Registered new tserver with Master: 46a4d5c5a09449ff96f32a0b33c601b3 (127.4.27.1:43453)
I20260812 14:54:20.357612 27379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:37917
I20260812 14:54:20.572760 28001 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:20.572901 28001 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:20.573163 27379 ts_manager.cc:194] Registered new tserver with Master: c1d6677e2d044d2e93d57e20461d35b5 (127.4.27.2:36629)
I20260812 14:54:20.573535 27379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:46893
Aug 12 14:54:20 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:20.743856 28259 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:20.744017 28259 heartbeater.cc:511] Master 127.4.27.62:33219 requested a full tablet report, sending...
I20260812 14:54:20.744323 27378 ts_manager.cc:194] Registered new tserver with Master: 09e7cfc8275948c39049cdebdc2a6323 (127.4.27.3:42249)
I20260812 14:54:20.744843 27378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:42405
I20260812 14:54:20.755205 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:54:20.763948 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.756372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.4.27.62:33219)
0812 14:54:20.756471 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.756473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.756481 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:20.756507 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.756508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.756520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.756607 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:20.757245 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.758027 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.759049 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.759364 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.759441 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.760876 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:20.760877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:20.760877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:20.760886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:20.762746 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:20.763138 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:20.763140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:20.763140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:20.763213 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:20.763408 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:20.763409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:20.763409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:20.763548 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:20.763655 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.763763 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.763852 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260812 14:54:20.766392 27378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:54:20.768754 27378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:54:20.777750 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.771778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36629 (local address 127.0.0.1:39996)
0812 14:54:20.772785 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.772823 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:20.772901 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:20.773176 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:20.773180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:20.773182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:20.773243 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.773245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.774848 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.774850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.776149 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.776152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.777462 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.777463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.777547 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.777559 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:54:20.777626 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:54:20.777676 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":85,"threads_started":1}
I20260812 14:54:20.778545 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.771784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42249 (local address 127.0.0.1:45742)
0812 14:54:20.774448 (+ 2664us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.774460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:20.774520 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:20.774811 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:20.774812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:20.774815 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:20.774871 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.774873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.776128 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.776129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.777264 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.777267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.778252 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.778253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.778331 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.778335 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:20.778438 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 14:54:20.778474 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":99,"threads_started":1}
I20260812 14:54:20.778877 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.772270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.4.27.3:42249)
0812 14:54:20.773031 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.773035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.774583 (+ 1548us) server_negotiation.cc:407] Connection header received
0812 14:54:20.774623 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.774625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.774643 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.774711 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:20.774950 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.776033 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.777358 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.778093 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.778167 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.778217 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.778758 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.778802 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":50,"threads_started":1}
I20260812 14:54:20.779712 28128 tablet_service.cc:1467] Processing CreateTablet for tablet 30758e0543584e40894941a17827be52 (DEFAULT_TABLE table=test-table [id=59699135aaa749dbb8b1a6a7b0b77094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:20.780061 28128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30758e0543584e40894941a17827be52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:20.782328 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.772021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43453 (local address 127.0.0.1:41872)
0812 14:54:20.778137 (+ 6116us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.778146 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:20.778184 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:20.778491 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:20.778492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:20.778494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:20.778523 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.778524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.780037 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.780038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.781107 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.781111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.782099 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.782113 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.782176 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.782180 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:20.782231 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:54:20.782266 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6104,"thread_start_us":72,"threads_started":1}
I20260812 14:54:20.782612 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.772247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.4.27.1:43453)
0812 14:54:20.772724 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.772729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.778242 (+ 5513us) server_negotiation.cc:407] Connection header received
0812 14:54:20.778282 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.778284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.778299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.778408 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:20.778602 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.779944 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.781208 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.781940 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.782009 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.782063 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.782489 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.782535 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":65,"threads_started":1}
I20260812 14:54:20.783363 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.772119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.4.27.2:36629)
0812 14:54:20.772405 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.772410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.772895 (+ 485us) server_negotiation.cc:407] Connection header received
0812 14:54:20.772979 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.772981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.772997 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.773075 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:20.773333 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.774752 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.776251 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.777292 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.777362 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.777422 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.783191 (+ 5769us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.783279 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":71,"threads_started":1}
I20260812 14:54:20.783268 27593 tablet_service.cc:1467] Processing CreateTablet for tablet 30758e0543584e40894941a17827be52 (DEFAULT_TABLE table=test-table [id=59699135aaa749dbb8b1a6a7b0b77094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:20.783597 27593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30758e0543584e40894941a17827be52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:20.783959 27882 tablet_service.cc:1467] Processing CreateTablet for tablet 30758e0543584e40894941a17827be52 (DEFAULT_TABLE table=test-table [id=59699135aaa749dbb8b1a6a7b0b77094]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:20.784353 27882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30758e0543584e40894941a17827be52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:20.788549 28321 tablet_bootstrap.cc:492] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: Bootstrap starting.
I20260812 14:54:20.789184 28321 tablet_bootstrap.cc:654] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:20.789469 28321 log.cc:824] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:20.789783 28319 tablet_bootstrap.cc:492] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Bootstrap starting.
I20260812 14:54:20.790521 28318 tablet_bootstrap.cc:492] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Bootstrap starting.
I20260812 14:54:20.790760 28321 tablet_bootstrap.cc:492] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: No bootstrap required, opened a new log
I20260812 14:54:20.790959 28321 ts_tablet_manager.cc:1397] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:54:20.791199 28318 tablet_bootstrap.cc:654] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:20.791571 28318 log.cc:824] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:20.793467 28319 tablet_bootstrap.cc:654] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:20.793756 28319 log.cc:824] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:20.794090 28321 raft_consensus.cc:348] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.794260 28321 raft_consensus.cc:374] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:20.794689 28319 tablet_bootstrap.cc:492] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: No bootstrap required, opened a new log
I20260812 14:54:20.794755 28319 ts_tablet_manager.cc:1397] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260812 14:54:20.797039 28319 raft_consensus.cc:348] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.797199 28319 raft_consensus.cc:374] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:20.797230 28319 raft_consensus.cc:729] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d6677e2d044d2e93d57e20461d35b5, State: Initialized, Role: FOLLOWER
I20260812 14:54:20.797307 28321 raft_consensus.cc:729] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46a4d5c5a09449ff96f32a0b33c601b3, State: Initialized, Role: FOLLOWER
I20260812 14:54:20.797465 28321 consensus_queue.cc:260] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.797716 28321 ts_tablet_manager.cc:1428] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260812 14:54:20.797755 27763 heartbeater.cc:503] Master 127.4.27.62:33219 was elected leader, sending a full tablet report...
I20260812 14:54:20.797879 28319 consensus_queue.cc:260] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.798130 28319 ts_tablet_manager.cc:1428] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260812 14:54:20.798193 28001 heartbeater.cc:503] Master 127.4.27.62:33219 was elected leader, sending a full tablet report...
W20260812 14:54:20.798818 28003 tablet.cc:2369] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:54:20.802965 28318 tablet_bootstrap.cc:492] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: No bootstrap required, opened a new log
I20260812 14:54:20.803036 28318 ts_tablet_manager.cc:1397] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260812 14:54:20.805524 28318 raft_consensus.cc:348] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.805675 28318 raft_consensus.cc:374] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:20.805707 28318 raft_consensus.cc:729] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e7cfc8275948c39049cdebdc2a6323, State: Initialized, Role: FOLLOWER
I20260812 14:54:20.805821 28318 consensus_queue.cc:260] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.806077 28318 ts_tablet_manager.cc:1428] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:54:20.806115 28259 heartbeater.cc:503] Master 127.4.27.62:33219 was elected leader, sending a full tablet report...
I20260812 14:54:20.814399 28327 raft_consensus.cc:482] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:20.814522 28327 raft_consensus.cc:504] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.814875 28327 leader_election.cc:288] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46a4d5c5a09449ff96f32a0b33c601b3 (127.4.27.1:43453), 09e7cfc8275948c39049cdebdc2a6323 (127.4.27.3:42249)
I20260812 14:54:20.818981 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.815073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:39097 (local address 127.4.27.1:43453)
0812 14:54:20.815115 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.815117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.815483 (+ 366us) server_negotiation.cc:407] Connection header received
0812 14:54:20.815518 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.815519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.815529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.815590 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:20.815741 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.816490 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.817967 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.818615 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.818685 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.818697 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.818874 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.818915 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:54:20.819002 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.815024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43453 (local address 127.4.27.2:39097)
0812 14:54:20.815405 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.815415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:20.815472 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:20.815599 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:20.815600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:20.815602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:20.815678 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.815680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.816585 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.816586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.817845 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.817849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.818717 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.818719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.818790 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.818796 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 14:54:20.818864 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:54:20.818907 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":98,"threads_started":1}
I20260812 14:54:20.819324 27668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30758e0543584e40894941a17827be52" candidate_uuid: "c1d6677e2d044d2e93d57e20461d35b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" is_pre_election: true
I20260812 14:54:20.819496 27668 raft_consensus.cc:2454] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d6677e2d044d2e93d57e20461d35b5 in term 0.
I20260812 14:54:20.819757 27800 leader_election.cc:302] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a4d5c5a09449ff96f32a0b33c601b3, c1d6677e2d044d2e93d57e20461d35b5; no voters:
I20260812 14:54:20.819923 28327 raft_consensus.cc:2781] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:54:20.820041 28327 raft_consensus.cc:482] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:54:20.820109 28327 raft_consensus.cc:3037] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:20.821030 28327 raft_consensus.cc:504] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.821379 27668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30758e0543584e40894941a17827be52" candidate_uuid: "c1d6677e2d044d2e93d57e20461d35b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46a4d5c5a09449ff96f32a0b33c601b3"
I20260812 14:54:20.821452 27668 raft_consensus.cc:3037] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:20.822320 27668 raft_consensus.cc:2454] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d6677e2d044d2e93d57e20461d35b5 in term 1.
I20260812 14:54:20.822563 27800 leader_election.cc:302] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46a4d5c5a09449ff96f32a0b33c601b3, c1d6677e2d044d2e93d57e20461d35b5; no voters:
I20260812 14:54:20.822804 28327 leader_election.cc:288] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [CANDIDATE]: Term 1 election: Requested vote from peers 46a4d5c5a09449ff96f32a0b33c601b3 (127.4.27.1:43453), 09e7cfc8275948c39049cdebdc2a6323 (127.4.27.3:42249)
I20260812 14:54:20.822883 28327 raft_consensus.cc:2781] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:20.822949 28327 raft_consensus.cc:686] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 1 LEADER]: Becoming Leader. State: Replica: c1d6677e2d044d2e93d57e20461d35b5, State: Running, Role: LEADER
I20260812 14:54:20.823058 28327 consensus_queue.cc:237] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } }
I20260812 14:54:20.823132 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.815340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:32863 (local address 127.4.27.3:42249)
0812 14:54:20.815376 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.815378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.819974 (+ 4596us) server_negotiation.cc:407] Connection header received
0812 14:54:20.820069 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.820070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.820082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.820119 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:20.820275 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.820919 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.822181 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.822751 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.822800 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.822809 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.823014 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.823055 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:54:20.823414 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.815273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42249 (local address 127.4.27.2:32863)
0812 14:54:20.819916 (+ 4643us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.819926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:20.820003 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:20.820179 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:20.820180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:20.820181 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:20.820212 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.820213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.821055 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.821055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.822057 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:20.822061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.822871 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.822872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:20.822934 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.822937 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:54:20.823309 (+ 372us) client_negotiation.cc:240] Negotiation successful
0812 14:54:20.823351 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4630,"thread_start_us":76,"threads_started":1}
I20260812 14:54:20.823907 28168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30758e0543584e40894941a17827be52" candidate_uuid: "c1d6677e2d044d2e93d57e20461d35b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e7cfc8275948c39049cdebdc2a6323" is_pre_election: true
I20260812 14:54:20.823930 27378 catalog_manager.cc:5495] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d6677e2d044d2e93d57e20461d35b5 (127.4.27.2). New cstate: current_term: 1 leader_uuid: "c1d6677e2d044d2e93d57e20461d35b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d6677e2d044d2e93d57e20461d35b5" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 } health_report { overall_health: UNKNOWN } } }
I20260812 14:54:20.824041 28168 raft_consensus.cc:2454] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1d6677e2d044d2e93d57e20461d35b5 in term 0.
I20260812 14:54:20.824177 28166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30758e0543584e40894941a17827be52" candidate_uuid: "c1d6677e2d044d2e93d57e20461d35b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e7cfc8275948c39049cdebdc2a6323"
I20260812 14:54:20.824234 28166 raft_consensus.cc:3037] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:20.825044 28166 raft_consensus.cc:2454] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d6677e2d044d2e93d57e20461d35b5 in term 1.
I20260812 14:54:20.837822 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.832781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.4.27.2:36629)
0812 14:54:20.832819 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:20.832820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:20.832827 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:54:20.832861 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:20.832862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:20.832881 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:20.832955 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:54:20.833076 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.833772 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.834861 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:20.835152 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:20.835227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:20.837312 (+ 2085us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:54:20.837433 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:54:20.837508 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:20.837734 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 14:54:20.837755 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:54:20.839831 28166 raft_consensus.cc:1261] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 1 FOLLOWER]: Refusing update from remote peer c1d6677e2d044d2e93d57e20461d35b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:20.839973 27668 raft_consensus.cc:1261] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3 [term 1 FOLLOWER]: Refusing update from remote peer c1d6677e2d044d2e93d57e20461d35b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:20.840313 28327 consensus_queue.cc:1035] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e7cfc8275948c39049cdebdc2a6323" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:20.840473 28327 consensus_queue.cc:1035] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46a4d5c5a09449ff96f32a0b33c601b3" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:54:20.840754 27764 tablet.cc:2369] T 30758e0543584e40894941a17827be52 P 46a4d5c5a09449ff96f32a0b33c601b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:54:20.845674 28342 mvcc.cc:205] Tried to move back new op lower bound from 7317694303597092864 to 7317694303532302336. Current Snapshot: MvccSnapshot[applied={T|T < 7317694303597092864}]
I20260812 14:54:20.849612 27379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:20.849701 27379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:20.850528 27379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65a22e608f924788896970311ef00900: Sending DeleteTablet for 3 replicas of tablet 30758e0543584e40894941a17827be52
I20260812 14:54:20.850979 28128 tablet_service.cc:1514] Processing DeleteTablet for tablet 30758e0543584e40894941a17827be52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:20 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:45742
I20260812 14:54:20.851403 27593 tablet_service.cc:1514] Processing DeleteTablet for tablet 30758e0543584e40894941a17827be52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:20 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:41872
I20260812 14:54:20.851984 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 27447
I20260812 14:54:20.860085 27882 tablet_service.cc:1514] Processing DeleteTablet for tablet 30758e0543584e40894941a17827be52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:20 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:39996
I20260812 14:54:20.860477 28355 tablet_replica.cc:330] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: stopping tablet replica
I20260812 14:54:20.860621 28355 raft_consensus.cc:2229] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:54:20.860762 28355 raft_consensus.cc:2258] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:20.864336 28355 ts_tablet_manager.cc:1905] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:20.866240 28355 ts_tablet_manager.cc:1918] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:20.866398 28355 log.cc:1196] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/30758e0543584e40894941a17827be52
I20260812 14:54:20.866747 28355 ts_tablet_manager.cc:1939] T 30758e0543584e40894941a17827be52 P 09e7cfc8275948c39049cdebdc2a6323: Deleting consensus metadata
I20260812 14:54:20.867174 27360 catalog_manager.cc:4841] TS 09e7cfc8275948c39049cdebdc2a6323 (127.4.27.3:42249): tablet 30758e0543584e40894941a17827be52 (table test-table [id=59699135aaa749dbb8b1a6a7b0b77094]) successfully deleted
I20260812 14:54:20.870494 28359 tablet_replica.cc:330] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: stopping tablet replica
I20260812 14:54:20.870687 28359 raft_consensus.cc:2229] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:54:20.870865 28359 raft_consensus.cc:2258] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 14:54:20.871435 27358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:43453 (error 108)
I20260812 14:54:20.871718 28359 ts_tablet_manager.cc:1905] T 30758e0543584e40894941a17827be52 P c1d6677e2d044d2e93d57e20461d35b5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:54:20.872097 28310 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:54:20.871848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43453 (local address 127.0.0.1:41882)
0812 14:54:20.871896 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.871907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:20.871943 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:20.872051 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43453: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:54:20.872385 27358 catalog_manager.cc:4584] TS 46a4d5c5a09449ff96f32a0b33c601b3 (127.4.27.1:43453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30758e0543584e40894941a17827be52: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43453: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:54:20.872901 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 27768
I20260812 14:54:20.894711 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 28006
I20260812 14:54:20.898389 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:20.898224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36629 (local address 127.0.0.1:40016)
0812 14:54:20.898280 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:20.898325 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.2:36629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Aug 12 14:54:20 dist-test-slave-6zbq krb5kdc[25559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for HTTP/127.4.27.20@KRBTEST.COM
I20260812 14:54:20.905416 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 27328
I20260812 14:54:20.919917 4204 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 14:54:21.263590 4204 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 14:54:21.263684 4204 mini_ranger.cc:66] Stopping Ranger...
I20260812 14:54:21.607481 4204 mini_ranger.cc:68] Stopped Ranger
2026-08-12 14:54:21.607 UTC [25613] LOG: received smart shutdown request
2026-08-12 14:54:21.611 UTC [25613] LOG: background worker "logical replication launcher" (PID 25633) exited with exit code 1
2026-08-12 14:54:21.611 UTC [25619] LOG: shutting down
2026-08-12 14:54:21.739 UTC [25613] LOG: database system is shut down
2026-08-12T14:54:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46167 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:54:21 dist-test-slave-6zbq krb5kdc[28378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:54:21 dist-test-slave-6zbq krb5kdc[28378](info): set up 2 sockets
Aug 12 14:54:21 dist-test-slave-6zbq krb5kdc[28378](info): commencing operation
krb5kdc: starting...
W20260812 14:54:23.017596 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:54:23.128756 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36615
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36615,127.4.27.61:46779,127.4.27.60:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:23.209761 28407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:23.209935 28407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:23.209955 28407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:23.211081 28407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:23.211122 28407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:23.211135 28407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:23.211145 28407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:23.211156 28407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:23.212659 28407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36615,127.4.27.61:46779,127.4.27.60:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:23.212873 28407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:23.213069 28407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:23.217828 28420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:23.217810 28421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:23.217810 28424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:23.217916 28407 server_base.cc:1034] running on GCE node
I20260812 14:54:23.218334 28407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:23.218652 28407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:23.219794 28407 hybrid_clock.cc:648] HybridClock initialized: now 1786546463219776 us; error 33 us; skew 500 ppm
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:23.222097 28407 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:23.222532 28407 webserver.cc:460] Webserver started at http://127.4.27.62:37665/ using document root <none> and password file <none>
I20260812 14:54:23.222741 28407 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:23.222788 28407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:23.222915 28407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:23.223875 28407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "07f719bf0ea9d9cfe9524fe2ae0e2c21"
server_key_iv: "64a1370ddbca0bb86c9cce6373b85ab8"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.224179 28407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "07f719bf0ea9d9cfe9524fe2ae0e2c21"
server_key_iv: "64a1370ddbca0bb86c9cce6373b85ab8"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.225343 28407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:54:23.226085 28439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:23.226235 28407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:23.226315 28407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "07f719bf0ea9d9cfe9524fe2ae0e2c21"
server_key_iv: "64a1370ddbca0bb86c9cce6373b85ab8"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.226418 28407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:23.241338 28407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:23.243289 28407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:23.243415 28407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:23.247020 28407 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:36615
I20260812 14:54:23.247051 28540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:36615 every 8 connection(s)
I20260812 14:54:23.247347 28407 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:23.248109 28541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:23.249526 28541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:54:23.253072 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 28407
I20260812 14:54:23.253189 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:54:23.253350 4204 external_mini_cluster.cc:1428] Setting key 2ddd33952483f3e5c37865c88424060b
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:23.258139 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.250093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36615 (local address 127.0.0.1:33932)
0812 14:54:23.250243 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.250280 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.250756 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.250974 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.250976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.251196 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.251336 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.251339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.252260 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.252261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.252923 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.252927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.253423 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.253423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.253490 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.253896 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.253904 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.255163 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.256851 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.256857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.256858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.257040 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.257308 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.257308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.257308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.257374 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.257754 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.257760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.257813 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.257881 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.257998 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260812 14:54:23.258225 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.250124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.4.27.62:36615)
0812 14:54:23.250287 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.250291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.250359 (+ 68us) server_negotiation.cc:407] Connection header received
0812 14:54:23.250870 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.250872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.250888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.250957 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.251409 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.252193 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.252997 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.253347 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.253449 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.255239 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.255246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.255247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.255252 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.256765 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.257116 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.257120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.257120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.257243 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.257425 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.257426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.257426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.257692 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.257777 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.257924 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.258079 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260812 14:54:23.258860 28541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:54:23.259121 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.259021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37198)
0812 14:54:23.259060 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.259101 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:54:23.259236 28445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
I20260812 14:54:23.259569 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.259492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37210)
0812 14:54:23.259525 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.259549 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 14:54:23.259660 28541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:46779: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.27.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab.
Entry for principal kudu/127.4.27.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab.
I20260812 14:54:23.276353 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:46779
--webserver_interface=127.4.27.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab
--principal=kudu/127.4.27.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36615,127.4.27.61:46779,127.4.27.60:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:23.308856 28541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } attempt: 1
I20260812 14:54:23.309316 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.309198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37222)
0812 14:54:23.309255 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.309289 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:54:23.309648 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.309562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37238)
0812 14:54:23.309598 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.309622 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:54:23.309742 28541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:46779: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
W20260812 14:54:23.354940 28552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:23.355125 28552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:23.355154 28552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:23.356233 28552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:23.356273 28552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:23.356298 28552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:23.356315 28552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:23.356331 28552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:23.357815 28552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36615,127.4.27.61:46779,127.4.27.60:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab
--principal=kudu/127.4.27.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.27.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:23.358040 28552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:23.358233 28552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:23.362579 28563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:23.362638 28560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:23.362655 28561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:23.362774 28552 server_base.cc:1034] running on GCE node
I20260812 14:54:23.363001 28552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:23.363265 28552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:23.364409 28552 hybrid_clock.cc:648] HybridClock initialized: now 1786546463364393 us; error 29 us; skew 500 ppm
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:23.366847 28552 init.cc:377] Logged in from keytab as kudu/127.4.27.61@KRBTEST.COM (short username kudu)
I20260812 14:54:23.367231 28552 webserver.cc:460] Webserver started at http://127.4.27.61:34613/ using document root <none> and password file <none>
I20260812 14:54:23.367426 28552 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:23.367472 28552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:23.367588 28552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:23.368556 28552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data/instance:
uuid: "5fcbd2d2a6c1440f968b778660af3425"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "83ff9ff51d87de53d452c8e1adc8830d"
server_key_iv: "938757f5ba03c5e4237c74dd16ca21dd"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.368870 28552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal/instance:
uuid: "5fcbd2d2a6c1440f968b778660af3425"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "83ff9ff51d87de53d452c8e1adc8830d"
server_key_iv: "938757f5ba03c5e4237c74dd16ca21dd"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.370230 28552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:23.370921 28575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:23.371068 28552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:23.371151 28552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal
uuid: "5fcbd2d2a6c1440f968b778660af3425"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "83ff9ff51d87de53d452c8e1adc8830d"
server_key_iv: "938757f5ba03c5e4237c74dd16ca21dd"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.371213 28552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:23.377861 28541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } attempt: 2
I20260812 14:54:23.378175 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.378066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37240)
0812 14:54:23.378117 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.378147 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:54:23.378523 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.378451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37252)
0812 14:54:23.378482 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.378506 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260812 14:54:23.378611 28541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:46779: Network error: Client connection negotiation failed: client connection to 127.4.27.61:46779: connect: Connection refused (error 111)
I20260812 14:54:23.379267 28552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:23.381192 28552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:23.381317 28552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:23.384826 28552 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.61:46779
I20260812 14:54:23.384864 28677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.61:46779 every 8 connection(s)
I20260812 14:54:23.385154 28552 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
I20260812 14:54:23.385910 28680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:23.387293 28680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:54:23.390498 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 28552
I20260812 14:54:23.390619 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-1/wal/instance
I20260812 14:54:23.390807 4204 external_mini_cluster.cc:1428] Setting key a9d5b5df37adf479fe78e2cb87e2a927
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:23.395560 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.387871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.4.27.62:36615)
0812 14:54:23.387922 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.387924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.388108 (+ 184us) server_negotiation.cc:407] Connection header received
0812 14:54:23.388512 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.388513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.388521 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.388569 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.389064 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.389523 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.390604 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.390854 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.390922 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.392912 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.392913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.392914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.392921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.394252 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.394638 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.394639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.394640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.394704 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.394918 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.394919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.394919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.395058 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.395143 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.395389 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.395473 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:54:23.395592 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.387845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36615 (local address 127.0.0.1:33934)
0812 14:54:23.387999 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.388036 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.388442 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.388620 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.388622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.388833 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.389002 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.389006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.389579 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.389579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.390512 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.390515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.390934 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.390935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.391060 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.391446 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.391454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.392826 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.394328 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.394333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.394335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.394567 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.394764 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.394764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.394765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.394852 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.395137 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.395144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.395268 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.395349 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.395456 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260812 14:54:23.396214 28680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.27.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.60@KRBTEST.COM" created.
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:54:23.405166 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.396423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37262)
0812 14:54:23.396465 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.396475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.396512 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.396701 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.396702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.396838 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.396896 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.396898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.397669 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.397670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.398453 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.398456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.398861 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.398862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.398938 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.399238 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.399243 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.402629 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.404194 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.404195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.404196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.404389 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.404557 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.404558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.404558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.404624 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.404848 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.404849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.404892 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.404959 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.405049 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:54:23.405187 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.396462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.4.27.61:46779)
0812 14:54:23.396579 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.396582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.396595 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:54:23.396612 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.396614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.396623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.396680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.396959 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.397596 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.398519 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.398789 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.398839 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.402714 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.402725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.402725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.402732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.404114 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.404457 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.404458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.404459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.404506 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.404679 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.404679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.404679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.404792 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.404845 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.404985 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.405101 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260812 14:54:23.405730 28680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:54:23.406019 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.405900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47712)
0812 14:54:23.405948 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.405997 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:54:23.406150 28580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
I20260812 14:54:23.406548 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.406454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47718)
0812 14:54:23.406496 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.406526 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:54:23.406644 28680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:46291: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab.
Entry for principal kudu/127.4.27.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab.
I20260812 14:54:23.414891 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:46291
--webserver_interface=127.4.27.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab
--principal=kudu/127.4.27.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36615,127.4.27.61:46779,127.4.27.60:46291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:23.455873 28680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } attempt: 1
I20260812 14:54:23.456436 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.456268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47728)
0812 14:54:23.456360 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.456404 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 14:54:23.456772 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.456697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47740)
0812 14:54:23.456732 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.456755 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:54:23.456872 28680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:46291: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
I20260812 14:54:23.469748 28541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } attempt: 3
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:54:23.476812 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.470122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37264)
0812 14:54:23.470184 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.470197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.470262 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.470429 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.470431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.470573 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.470651 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.470652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.471609 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.471611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.472123 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.472124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.472598 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.472599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.472650 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.473135 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.473140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.474450 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.476025 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.476026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.476026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.476174 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.476349 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.476349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.476349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.476394 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.476606 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.476607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.476640 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.476683 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.476750 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260812 14:54:23.476878 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.470198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.4.27.61:46779)
0812 14:54:23.470263 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.470266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.470301 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:54:23.470329 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.470330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.470367 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.470423 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.470713 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.471496 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.472206 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.472524 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.472595 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.474515 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.474517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.474517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.474525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.475953 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.476233 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.476234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.476234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.476293 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.476437 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.476438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.476438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.476558 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.476618 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.476706 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.476789 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:54:23.477200 28541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:54:23.477506 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.477411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47746)
0812 14:54:23.477453 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.477479 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:54:23.477803 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.477739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47748)
0812 14:54:23.477770 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.477790 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 14:54:23.477881 28541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:46291: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
W20260812 14:54:23.498708 28692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:23.498926 28692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:23.498955 28692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:23.500242 28692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:23.500299 28692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:23.500313 28692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:23.500325 28692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:23.500335 28692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:23.501820 28692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36615,127.4.27.61:46779,127.4.27.60:46291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab
--principal=kudu/127.4.27.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:46291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.27.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:23.502003 28692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:23.502172 28692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:23.506501 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:23.506623 28692 server_base.cc:1034] running on GCE node
W20260812 14:54:23.506505 28709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:23.506485 28705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:23.506927 28692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:23.507225 28692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:23.508360 28692 hybrid_clock.cc:648] HybridClock initialized: now 1786546463508350 us; error 24 us; skew 500 ppm
I20260812 14:54:23.509002 28541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } attempt: 1
I20260812 14:54:23.509346 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.509221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47754)
0812 14:54:23.509276 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.509311 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:54:23.509647 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.509563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47770)
0812 14:54:23.509599 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.509626 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:54:23.509743 28541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:46291: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:23.510879 28692 init.cc:377] Logged in from keytab as kudu/127.4.27.60@KRBTEST.COM (short username kudu)
I20260812 14:54:23.511250 28692 webserver.cc:460] Webserver started at http://127.4.27.60:43041/ using document root <none> and password file <none>
I20260812 14:54:23.511453 28692 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:23.511497 28692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:23.511613 28692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:23.512569 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data/instance:
uuid: "696f942dca014a9b856857a523d9b3ac"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "cdd53d6421cf64a6486369df1072a918"
server_key_iv: "7ba92ab2cced8cb7ac2ef44c8a579408"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.512877 28692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal/instance:
uuid: "696f942dca014a9b856857a523d9b3ac"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "cdd53d6421cf64a6486369df1072a918"
server_key_iv: "7ba92ab2cced8cb7ac2ef44c8a579408"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.514024 28692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:54:23.514691 28717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:23.514832 28692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:23.514909 28692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal
uuid: "696f942dca014a9b856857a523d9b3ac"
format_stamp: "Formatted at 2026-08-12 14:54:23 on dist-test-slave-6zbq"
server_key: "cdd53d6421cf64a6486369df1072a918"
server_key_iv: "7ba92ab2cced8cb7ac2ef44c8a579408"
server_key_version: "encryptionkey@0"
I20260812 14:54:23.514973 28692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:23.525038 28680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } attempt: 2
I20260812 14:54:23.525411 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.525309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47782)
0812 14:54:23.525358 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.525387 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 14:54:23.525695 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.525619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47798)
0812 14:54:23.525654 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.525677 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 14:54:23.525767 28680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:46291: Network error: Client connection negotiation failed: client connection to 127.4.27.60:46291: connect: Connection refused (error 111)
I20260812 14:54:23.543586 28692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:23.545172 28692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:23.545310 28692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:23.548954 28692 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.60:46291
I20260812 14:54:23.548986 28798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.60:46291 every 8 connection(s)
I20260812 14:54:23.549299 28692 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
I20260812 14:54:23.549567 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 28692
I20260812 14:54:23.549685 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786546313808590-4204-0/minicluster-data/master-2/wal/instance
I20260812 14:54:23.549907 4204 external_mini_cluster.cc:1428] Setting key e7ff174e0be54e8c624943f53a588332
I20260812 14:54:23.550267 28799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:23.552440 28799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:23.560073 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.550414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.4.27.60:46291)
0812 14:54:23.550603 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.550612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.550643 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:54:23.551369 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.551378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.551442 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.551596 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.552132 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.553378 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.554264 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.554819 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.554925 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.557186 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.557193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.557193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.557199 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.558597 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.558945 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.558948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.558949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.559038 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.559247 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.559248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.559249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.559537 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.559667 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.559766 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.559876 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260812 14:54:23.560212 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.553106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.4.27.62:36615)
0812 14:54:23.553153 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.553155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.553260 (+ 105us) server_negotiation.cc:407] Connection header received
0812 14:54:23.553322 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.553323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.553331 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.553374 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.553774 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.554464 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.555408 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.555690 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.555766 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.557672 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.557673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.557673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.557680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.559202 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.559523 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.559524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.559524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.559578 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.559737 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.559738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.559738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.559840 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.559894 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.560052 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.560125 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:54:23.560225 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.553027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:36615 (local address 127.0.0.1:33950)
0812 14:54:23.553166 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.553201 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.553266 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.553392 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.553394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.553643 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.553704 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.553706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.554531 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.554532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.555328 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.555331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.555768 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.555769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.555827 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.556196 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.556204 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.557595 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.559269 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.559276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.559278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.559459 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.559630 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.559630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.559631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.559687 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.559892 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.559899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.559951 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.560021 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.560124 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260812 14:54:23.560787 28799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:23.566505 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.561007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:46779 (local address 127.0.0.1:37272)
0812 14:54:23.561040 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.561049 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.561078 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.561205 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.561206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.561330 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.561355 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.561357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.561919 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.561920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.562424 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.562426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.562738 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.562739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.562799 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.563164 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.563168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.564178 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.565652 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.565653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.565653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.565808 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.565986 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.565986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.565986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.566028 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.566229 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.566230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.566270 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.566334 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.566430 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260812 14:54:23.566540 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.561057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.4.27.61:46779)
0812 14:54:23.561117 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.561119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.561127 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:23.561150 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.561151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.561160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.561203 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.561418 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.561855 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.562496 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.562687 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.562735 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.564255 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.564256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.564256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.564262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.565586 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.565874 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.565875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.565875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.565930 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.566066 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.566067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.566067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.566174 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.566219 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.566383 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.566455 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:54:23.566860 28799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:54:23.567230 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.560861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.4.27.62:36615)
0812 14:54:23.560890 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.560892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.560899 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:54:23.560941 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.560941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.560949 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.560997 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.561430 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.561849 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.562513 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.562693 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.562741 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.564825 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.564827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.564827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.564834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.566027 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.566336 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.566358 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.566359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.566412 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.566628 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.566629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.566630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.566735 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.566873 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.567078 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.567153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:54:23.572618 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.567064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47816)
0812 14:54:23.567134 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.567143 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.567188 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.567366 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.567367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.567493 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.567521 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.567522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.568166 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.568167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.568688 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.568690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.569001 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.569001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.569048 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.569385 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.569388 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.570389 (+ 1001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.571663 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.571664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.571665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.571861 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.572047 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.572048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.572048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.572111 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.572323 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.572324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.572360 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.572419 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.572508 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 14:54:23.572669 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.567100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.4.27.60:46291)
0812 14:54:23.567132 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.567134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.567174 (+ 40us) server_negotiation.cc:407] Connection header received
0812 14:54:23.567269 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.567270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.567278 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.567333 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.567579 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.568100 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.568762 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.568947 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.568991 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.570453 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.570455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.570455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.570461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.571593 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.571935 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.571936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.571936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.571983 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.572171 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.572171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.572172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.572256 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.572305 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.572445 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.572555 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":17}
I20260812 14:54:23.574086 28799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: Bootstrap starting.
I20260812 14:54:23.574954 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.568017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.4.27.61:46779)
0812 14:54:23.568050 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.568051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.568059 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:23.568098 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.568098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.568105 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.568149 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.568556 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.569079 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.569773 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.569951 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.570002 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.572238 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.572239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.572239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.572246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.573836 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.574164 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.574166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.574166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.574225 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.574448 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.574449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.574449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.574585 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.574772 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.574809 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.574879 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:54:23.575066 28799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:23.575366 28799 log.cc:824] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:23.576094 28799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: No bootstrap required, opened a new log
I20260812 14:54:23.576287 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:54:23.578099 28799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.578377 28799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:23.578480 28799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696f942dca014a9b856857a523d9b3ac, State: Initialized, Role: FOLLOWER
I20260812 14:54:23.578692 28799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.579077 28799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:23.580480 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.580647 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:54:23.581866 28860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:54:23.581913 28860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:54:23.581374 28692 master_runner.cc:186] Error getting master registration for 127.4.27.62:36615: OK, instance_id { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" instance_seqno: 1786546463246023 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:54:23.584091 28692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:54:23.584997 28541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } attempt: 2
I20260812 14:54:23.585598 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.577884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.4.27.60:46291)
0812 14:54:23.577917 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.577919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.577994 (+ 75us) server_negotiation.cc:407] Connection header received
0812 14:54:23.578157 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.578158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.578167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.578211 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.579006 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.579675 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.580650 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.580934 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.580969 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.582803 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.582805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.582805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.582811 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.584263 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.584661 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.584662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.584662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.584719 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.584914 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.584915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.584915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.585057 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.585139 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.585452 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.585525 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:54:23.587803 28688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.577710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.4.27.61:46779)
0812 14:54:23.577774 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.577775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.577970 (+ 195us) server_negotiation.cc:407] Connection header received
0812 14:54:23.578051 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.578052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.578061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.578103 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.578913 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.579608 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.580713 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.580979 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.581013 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.583820 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.583822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.583822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.583829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.585498 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.586556 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.586557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.586558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.586624 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.586822 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.586823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.586823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.586934 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.587435 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.587651 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.587724 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 14:54:23.587832 28546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.577666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.4.27.62:36615)
0812 14:54:23.577749 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.577750 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.577897 (+ 147us) server_negotiation.cc:407] Connection header received
0812 14:54:23.578118 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.578119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.578128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.578188 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.578827 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.579461 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.580633 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.580871 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.580905 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.584772 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.584774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.584774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.584781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.586740 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.587074 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.587075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.587075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.587146 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.587344 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.587345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.587346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.587457 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.587517 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.587694 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.587758 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:54:23.591907 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.585394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.4.27.60:46291)
0812 14:54:23.585667 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.585668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.585676 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:23.585692 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.585693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.585701 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.585751 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.586122 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.586825 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.587760 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.588034 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.588089 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.589781 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.589782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.589782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.589788 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.590958 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.591327 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.591328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.591328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.591372 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.591534 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.591535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.591535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.591625 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.591668 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.591807 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.591856 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":82,"threads_started":1}
I20260812 14:54:23.591958 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.585201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47838)
0812 14:54:23.585289 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.585298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.585334 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.585872 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.585873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.586008 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.586044 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.586045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.586917 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.586918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.587666 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.587669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.588134 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.588135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.588200 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.588538 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.588543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.589703 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.591034 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.591035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.591035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.591256 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.591426 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.591427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.591427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.591488 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.591678 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.591679 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.591719 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.591793 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.591872 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 14:54:23.593413 28541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82: Bootstrap starting.
I20260812 14:54:23.594061 28541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:23.594396 28541 log.cc:824] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:23.595132 28541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82: No bootstrap required, opened a new log
I20260812 14:54:23.596999 28541 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.597131 28541 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:23.597179 28541 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcf5e2ec4a54ff5a812e377ebde0e82, State: Initialized, Role: FOLLOWER
I20260812 14:54:23.597285 28541 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.597508 28866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.597556 28541 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:23.597600 28866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:54:23.599470 28898 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:54:23.599572 28898 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:54:23.599924 28407 master_runner.cc:186] Error getting master registration for 127.4.27.61:46779: OK, instance_id { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" instance_seqno: 1786546463383884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:54:23.601637 28407 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:54:23.616889 28680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } attempt: 3
Aug 12 14:54:23 dist-test-slave-6zbq krb5kdc[28378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:54:23.623083 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.617155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.4.27.60:46291)
0812 14:54:23.617186 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:23.617188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:23.617196 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:23.617256 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:23.617257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:23.617266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:23.617304 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:23.617561 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.618034 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.618814 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.618989 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.619041 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.620651 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:23.620652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:23.620653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:23.620660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:23.621889 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.622331 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.622332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.622332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.622412 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:23.622636 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:23.622637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:23.622637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:23.622736 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:23.622789 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:23.622934 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 14:54:23.622996 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:54:23.623092 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:23.617107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:46291 (local address 127.0.0.1:47854)
0812 14:54:23.617153 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:23.617163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:23.617200 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:23.617313 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:23.617314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:23.617455 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:23.617498 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.617500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.618114 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.618115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.618746 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:23.618748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:23.619046 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:23.619047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:23.619101 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:23.619478 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:23.619483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:23.620574 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:23.621967 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.621968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.621968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.622178 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.622490 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:23.622491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:23.622491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:23.622556 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:23.622799 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:23.622800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:23.622843 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:54:23.622902 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:54:23.623009 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:54:23.624575 28680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: Bootstrap starting.
I20260812 14:54:23.625159 28680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:23.625417 28680 log.cc:824] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:23.626091 28680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: No bootstrap required, opened a new log
I20260812 14:54:23.627482 28680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.627588 28680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:23.627621 28680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fcbd2d2a6c1440f968b778660af3425, State: Initialized, Role: FOLLOWER
I20260812 14:54:23.627753 28680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.627990 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.628048 28680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:23.628119 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:54:23.629815 28922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:54:23.629913 28922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260812 14:54:23.632063 28592 tablet.cc:2369] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:54:23.632983 28552 master_runner.cc:263] No leader master found from master 5fcbd2d2a6c1440f968b778660af3425
I20260812 14:54:23.633028 28552 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:54:23.729902 28866 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:23.730028 28866 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.730268 28866 leader_election.cc:288] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fcbd2d2a6c1440f968b778660af3425 (127.4.27.61:46779), 696f942dca014a9b856857a523d9b3ac (127.4.27.60:46291)
I20260812 14:54:23.730762 28760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696f942dca014a9b856857a523d9b3ac" is_pre_election: true
I20260812 14:54:23.730916 28760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbcf5e2ec4a54ff5a812e377ebde0e82 in term 0.
I20260812 14:54:23.731139 28442 leader_election.cc:302] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696f942dca014a9b856857a523d9b3ac, dbcf5e2ec4a54ff5a812e377ebde0e82; no voters:
I20260812 14:54:23.731423 28866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:54:23.731484 28866 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:54:23.731508 28866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:23.732373 28866 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.732494 28866 leader_election.cc:288] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [CANDIDATE]: Term 1 election: Requested vote from peers 5fcbd2d2a6c1440f968b778660af3425 (127.4.27.61:46779), 696f942dca014a9b856857a523d9b3ac (127.4.27.60:46291)
I20260812 14:54:23.732813 28760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "696f942dca014a9b856857a523d9b3ac"
I20260812 14:54:23.732889 28760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:23.733727 28760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbcf5e2ec4a54ff5a812e377ebde0e82 in term 1.
I20260812 14:54:23.733934 28442 leader_election.cc:302] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696f942dca014a9b856857a523d9b3ac, dbcf5e2ec4a54ff5a812e377ebde0e82; no voters:
I20260812 14:54:23.734808 28630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcbd2d2a6c1440f968b778660af3425" is_pre_election: true
I20260812 14:54:23.734946 28630 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbcf5e2ec4a54ff5a812e377ebde0e82 in term 0.
I20260812 14:54:23.735060 28629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fcbd2d2a6c1440f968b778660af3425"
I20260812 14:54:23.735107 28629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:23.735910 28629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbcf5e2ec4a54ff5a812e377ebde0e82 in term 1.
I20260812 14:54:23.736385 28866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:23.736503 28866 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [term 1 LEADER]: Becoming Leader. State: Replica: dbcf5e2ec4a54ff5a812e377ebde0e82, State: Running, Role: LEADER
I20260812 14:54:23.736585 28866 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } }
I20260812 14:54:23.736960 28866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcf5e2ec4a54ff5a812e377ebde0e82. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.737027 28866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:23.737327 28927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:23.737553 28927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:23.739770 28760 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [term 1 FOLLOWER]: Refusing update from remote peer dbcf5e2ec4a54ff5a812e377ebde0e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:23.740018 28629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [term 1 FOLLOWER]: Refusing update from remote peer dbcf5e2ec4a54ff5a812e377ebde0e82: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:23.740466 28866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:23.740597 28866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:23.741863 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcf5e2ec4a54ff5a812e377ebde0e82. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.741957 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:54:23.742951 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcf5e2ec4a54ff5a812e377ebde0e82. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.743049 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:54:23.744330 28866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.744405 28866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:23.744786 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.744861 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:54:23.745106 28930 mvcc.cc:205] Tried to move back new op lower bound from 7317694315474472960 to 7317694315465502720. Current Snapshot: MvccSnapshot[applied={T|T < 7317694315474472960}]
I20260812 14:54:23.746102 28936 mvcc.cc:205] Tried to move back new op lower bound from 7317694315474472960 to 7317694315465502720. Current Snapshot: MvccSnapshot[applied={T|T < 7317694315474472960}]
I20260812 14:54:23.746789 28866 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.746872 28866 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:23.747380 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbcf5e2ec4a54ff5a812e377ebde0e82" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36615 } } peers { permanent_uuid: "5fcbd2d2a6c1440f968b778660af3425" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 46779 } } peers { permanent_uuid: "696f942dca014a9b856857a523d9b3ac" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 46291 } } }
I20260812 14:54:23.747476 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:54:23.748160 28932 mvcc.cc:205] Tried to move back new op lower bound from 7317694315474472960 to 7317694315465502720. Current Snapshot: MvccSnapshot[applied={T|T < 7317694315474472960}]
I20260812 14:54:23.748746 28927 catalog_manager.cc:1347] Generated new cluster ID: ee3ab55601ca46c28b4eb637011b5ba4
I20260812 14:54:23.748798 28927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:54:23.776482 28927 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:54:23.777230 28927 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:23.781976 28927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcf5e2ec4a54ff5a812e377ebde0e82: Generated new TSK 0
I20260812 14:54:23.782282 28927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:54:24.582314 28860 catalog_manager.cc:1259] Loaded cluster ID: ee3ab55601ca46c28b4eb637011b5ba4
I20260812 14:54:24.582417 28860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: loading cluster ID for follower catalog manager: success
I20260812 14:54:24.583218 28860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: acquiring CA information for follower catalog manager: success
I20260812 14:54:24.583503 28860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 696f942dca014a9b856857a523d9b3ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:54:24.630362 28922 catalog_manager.cc:1259] Loaded cluster ID: ee3ab55601ca46c28b4eb637011b5ba4
I20260812 14:54:24.630426 28922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: loading cluster ID for follower catalog manager: success
I20260812 14:54:24.631206 28922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: acquiring CA information for follower catalog manager: success
I20260812 14:54:24.631479 28922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5fcbd2d2a6c1440f968b778660af3425: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:54:24.832326 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 28407
I20260812 14:54:24.846544 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 28552
I20260812 14:54:24.854684 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 28692
2026-08-12T14:54:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3026 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:54:24 dist-test-slave-6zbq krb5kdc[28956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:54:24 dist-test-slave-6zbq krb5kdc[28956](info): set up 2 sockets
Aug 12 14:54:24 dist-test-slave-6zbq krb5kdc[28956](info): commencing operation
krb5kdc: starting...
W20260812 14:54:26.033068 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:54:26 dist-test-slave-6zbq krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:54:26.149222 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36157
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:38997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:26.229635 28982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:26.229844 28982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:26.229874 28982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:26.231017 28982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:26.231058 28982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:26.231079 28982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:26.231094 28982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:26.231112 28982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:26.232576 28982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:38997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:26.232808 28982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:26.233008 28982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:26.238013 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:26.238068 28993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:26.238022 28990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:26.238073 28982 server_base.cc:1034] running on GCE node
I20260812 14:54:26.238464 28982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:26.238719 28982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:26.239859 28982 hybrid_clock.cc:648] HybridClock initialized: now 1786546466239845 us; error 31 us; skew 500 ppm
Aug 12 14:54:26 dist-test-slave-6zbq krb5kdc[28956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:26.242559 28982 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:26.242985 28982 webserver.cc:460] Webserver started at http://127.4.27.62:40033/ using document root <none> and password file <none>
I20260812 14:54:26.243189 28982 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:26.243238 28982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:26.243355 28982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:26.244354 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1"
format_stamp: "Formatted at 2026-08-12 14:54:26 on dist-test-slave-6zbq"
server_key: "f7ffed94be597702cff16a840f7e6061"
server_key_iv: "a87808a0ce014c032855d68191d41c0a"
server_key_version: "encryptionkey@0"
I20260812 14:54:26.244678 28982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1"
format_stamp: "Formatted at 2026-08-12 14:54:26 on dist-test-slave-6zbq"
server_key: "f7ffed94be597702cff16a840f7e6061"
server_key_iv: "a87808a0ce014c032855d68191d41c0a"
server_key_version: "encryptionkey@0"
I20260812 14:54:26.245860 28982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:54:26.246528 29007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:26.246671 28982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:26.246754 28982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1"
format_stamp: "Formatted at 2026-08-12 14:54:26 on dist-test-slave-6zbq"
server_key: "f7ffed94be597702cff16a840f7e6061"
server_key_iv: "a87808a0ce014c032855d68191d41c0a"
server_key_version: "encryptionkey@0"
I20260812 14:54:26.246821 28982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:26.256601 28982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:26.258147 28982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:26.258271 28982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:26.262004 28982 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:36157
I20260812 14:54:26.262048 29109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:36157 every 8 connection(s)
I20260812 14:54:26.262398 28982 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:26.263288 29111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:26.264032 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 28982
I20260812 14:54:26.264119 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:54:26.264304 4204 external_mini_cluster.cc:1428] Setting key ddd5c7be94735d28e5db40ae25544a4b
I20260812 14:54:26.266147 29111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1: Bootstrap starting.
I20260812 14:54:26.266783 29111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:26.267064 29111 log.cc:824] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:26.267812 29111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1: No bootstrap required, opened a new log
I20260812 14:54:26.270054 29111 raft_consensus.cc:348] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36157 } }
I20260812 14:54:26.270231 29111 raft_consensus.cc:374] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:26.270267 29111 raft_consensus.cc:729] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24b16a29fcd4dbbae9daa2b3ba987d1, State: Initialized, Role: FOLLOWER
I20260812 14:54:26.270421 29111 consensus_queue.cc:260] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36157 } }
I20260812 14:54:26.270493 29111 raft_consensus.cc:388] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:26.270514 29111 raft_consensus.cc:482] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:26.270558 29111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:54:26 dist-test-slave-6zbq krb5kdc[28956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:26.271096 29111 raft_consensus.cc:504] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36157 } }
I20260812 14:54:26.271188 29111 leader_election.cc:302] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c24b16a29fcd4dbbae9daa2b3ba987d1; no voters:
I20260812 14:54:26.271358 29111 leader_election.cc:288] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:54:26.271404 29119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:26.271600 29111 sys_catalog.cc:564] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:26.271818 29119 raft_consensus.cc:686] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [term 1 LEADER]: Becoming Leader. State: Replica: c24b16a29fcd4dbbae9daa2b3ba987d1, State: Running, Role: LEADER
I20260812 14:54:26.271961 29119 consensus_queue.cc:237] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36157 } }
I20260812 14:54:26.272382 29120 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36157 } } }
I20260812 14:54:26.272461 29120 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:26.272606 29123 sys_catalog.cc:455] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c24b16a29fcd4dbbae9daa2b3ba987d1. Latest consensus state: current_term: 1 leader_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c24b16a29fcd4dbbae9daa2b3ba987d1" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36157 } } }
I20260812 14:54:26.272665 29123 sys_catalog.cc:458] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:26.272969 29133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:26.273416 29133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:26.275041 29117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:26.264756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.4.27.62:36157)
0812 14:54:26.264944 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:26.264952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:26.264976 (+ 24us) server_negotiation.cc:407] Connection header received
0812 14:54:26.265400 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:26.265407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:26.265452 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:26.265570 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:26.266214 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:26.267217 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:26.268191 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:26.268592 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:26.268668 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:26.271231 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:26.271235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:26.271236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:26.271248 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:26.273130 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:26.273491 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:26.273494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:26.273495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:26.273593 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:26.273809 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:26.273810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:26.273810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:26.274110 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:26.274230 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:26.274789 (+ 559us) server_negotiation.cc:299] Negotiation successful
0812 14:54:26.274903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260812 14:54:26.275874 29133 catalog_manager.cc:1347] Generated new cluster ID: d86ea931204d420cb8006928611529a2
I20260812 14:54:26.275923 29133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:54:26.276031 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:54:26.288842 29133 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:54:26.289516 29133 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:26.297679 29133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c24b16a29fcd4dbbae9daa2b3ba987d1: Generated new TSK 0
I20260812 14:54:26.297851 29133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:54:26.330487 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 28982
2026-08-12T14:54:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1479 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:54:26 dist-test-slave-6zbq krb5kdc[29156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:54:26 dist-test-slave-6zbq krb5kdc[29156](info): set up 2 sockets
Aug 12 14:54:26 dist-test-slave-6zbq krb5kdc[29156](info): commencing operation
krb5kdc: starting...
W20260812 14:54:27.640185 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:54:27 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:54:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:54:27.745903 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42449
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:27.824456 29177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:27.824631 29177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:27.824651 29177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:27.825696 29177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:27.825728 29177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:27.825742 29177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:27.825752 29177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:27.825762 29177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:27.827169 29177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:27.827355 29177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:27.827517 29177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:27.831835 29186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:27.831843 29185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:27.832010 29177 server_base.cc:1034] running on GCE node
W20260812 14:54:27.832036 29189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:27.832297 29177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:27.832571 29177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:27.833707 29177 hybrid_clock.cc:648] HybridClock initialized: now 1786546467833693 us; error 29 us; skew 500 ppm
Aug 12 14:54:27 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:27.835937 29177 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:27.836349 29177 webserver.cc:460] Webserver started at http://127.4.27.62:35257/ using document root <none> and password file <none>
I20260812 14:54:27.836557 29177 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:27.836603 29177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:27.836719 29177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:27.837652 29177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "40a77d4c5e9d4d50bb148720dceff5bd"
format_stamp: "Formatted at 2026-08-12 14:54:27 on dist-test-slave-6zbq"
server_key: "8b27bb6ee5e989f73dce08758807ffa6"
server_key_iv: "1438787139a603a8e2e383784a273774"
server_key_version: "encryptionkey@0"
I20260812 14:54:27.837958 29177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "40a77d4c5e9d4d50bb148720dceff5bd"
format_stamp: "Formatted at 2026-08-12 14:54:27 on dist-test-slave-6zbq"
server_key: "8b27bb6ee5e989f73dce08758807ffa6"
server_key_iv: "1438787139a603a8e2e383784a273774"
server_key_version: "encryptionkey@0"
I20260812 14:54:27.839183 29177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:54:27.839834 29212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:27.839972 29177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:54:27.840054 29177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "40a77d4c5e9d4d50bb148720dceff5bd"
format_stamp: "Formatted at 2026-08-12 14:54:27 on dist-test-slave-6zbq"
server_key: "8b27bb6ee5e989f73dce08758807ffa6"
server_key_iv: "1438787139a603a8e2e383784a273774"
server_key_version: "encryptionkey@0"
I20260812 14:54:27.840111 29177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:27.864641 29177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:27.866137 29177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:27.866256 29177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:27.869915 29177 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:42449
I20260812 14:54:27.869969 29308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:42449 every 8 connection(s)
I20260812 14:54:27.870247 29177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:27.871104 29310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:27.873098 29310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap starting.
I20260812 14:54:27.873621 29310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:27.873863 29310 log.cc:824] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:27.874475 29310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: No bootstrap required, opened a new log
I20260812 14:54:27.875937 29310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:27.876060 29310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:27.876091 29310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a77d4c5e9d4d50bb148720dceff5bd, State: Initialized, Role: FOLLOWER
I20260812 14:54:27.876214 29310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:27.876281 29310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:27.876310 29310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:27.876355 29310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:27.877059 29310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:27.877151 29310 leader_election.cc:302] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a77d4c5e9d4d50bb148720dceff5bd; no voters:
I20260812 14:54:27.877305 29310 leader_election.cc:288] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:54:27.877342 29314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:27.877480 29314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 LEADER]: Becoming Leader. State: Replica: 40a77d4c5e9d4d50bb148720dceff5bd, State: Running, Role: LEADER
I20260812 14:54:27.877532 29310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:27.877592 29314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:27.877931 29315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } } }
I20260812 14:54:27.877952 29317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a77d4c5e9d4d50bb148720dceff5bd. Latest consensus state: current_term: 1 leader_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } } }
I20260812 14:54:27.878050 29315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:27.878139 29317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:27.878520 29334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:27.878917 29334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:27.880121 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 29177
I20260812 14:54:27.880220 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:54:27.880376 29334 catalog_manager.cc:1347] Generated new cluster ID: 86c1dacef716448bad683e2b373a135d
I20260812 14:54:27.880424 4204 external_mini_cluster.cc:1428] Setting key a10d9144cfc3a3dd17e4225fa22dd58c
I20260812 14:54:27.880456 29334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:54:27 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:27.892396 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:27.880993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.4.27.62:42449)
0812 14:54:27.882449 (+ 1456us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:27.882458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:27.882488 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:54:27.882589 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:27.882596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:27.882644 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:27.882759 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:27.883489 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:27.884792 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:27.885855 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:27.886460 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:27.886557 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:27.889086 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:27.889090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:27.889091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:27.889102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:27.890923 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:27.891265 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:27.891268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:27.891268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:27.891366 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:27.891537 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:27.891537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:27.891538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:27.891831 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:27.891944 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:27.892050 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 14:54:27.892182 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":68,"threads_started":1}
I20260812 14:54:27.905077 29334 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:54:27.905773 29334 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:27.928579 29334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Generated new TSK 0
I20260812 14:54:27.928740 29334 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:54:27.961799 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42449
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:28.040217 29364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:28.040393 29364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:28.040416 29364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:28.041505 29364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:28.041545 29364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:28.041569 29364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:54:28.043051 29364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:28.043262 29364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:28.043460 29364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:28.047888 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.047897 29374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.047885 29377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.048188 29364 server_base.cc:1034] running on GCE node
I20260812 14:54:28.048360 29364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:28.048571 29364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:28.049701 29364 hybrid_clock.cc:648] HybridClock initialized: now 1786546468049685 us; error 33 us; skew 500 ppm
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:28.052086 29364 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:54:28.052486 29364 webserver.cc:460] Webserver started at http://127.4.27.1:35417/ using document root <none> and password file <none>
I20260812 14:54:28.052673 29364 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:28.052711 29364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:28.052803 29364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:28.053676 29364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "aa575bd50168442e99f224acaba4b49c"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "62a20c86b186ec3a40aefc2c2d3eac98"
server_key_iv: "6d69a47af94a8135cfbfc01509b48fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.053946 29364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "aa575bd50168442e99f224acaba4b49c"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "62a20c86b186ec3a40aefc2c2d3eac98"
server_key_iv: "6d69a47af94a8135cfbfc01509b48fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.055105 29364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:54:28.055702 29391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.055871 29364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:54:28.055950 29364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "aa575bd50168442e99f224acaba4b49c"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "62a20c86b186ec3a40aefc2c2d3eac98"
server_key_iv: "6d69a47af94a8135cfbfc01509b48fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.056005 29364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:28.097921 29364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:28.099475 29364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:28.099574 29364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:28.099732 29364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:28.099967 29364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:28.099994 29364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.100013 29364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:28.100030 29364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.105391 29364 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37679
I20260812 14:54:28.105424 29611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37679 every 8 connection(s)
I20260812 14:54:28.105715 29364 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:54:28.106194 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 29364
I20260812 14:54:28.106287 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:54:28.106509 4204 external_mini_cluster.cc:1428] Setting key 488826ac9bacc6106a84d606071486b2
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:28.114917 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.106381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:41389 (local address 127.4.27.62:42449)
0812 14:54:28.106439 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.106442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.106666 (+ 224us) server_negotiation.cc:407] Connection header received
0812 14:54:28.107215 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.107215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.107224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.107262 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.108369 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.108894 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.109842 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.110048 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.110103 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.112278 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.112280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.112280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.112289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.113686 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.114031 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.114032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.114032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.114094 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.114289 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.114290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.114290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.114444 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.114514 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.114735 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.114819 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:54:28.114944 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.106329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:41389)
0812 14:54:28.106525 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.106572 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.107101 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.107340 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.107344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.108032 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:28.108289 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.108295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.108986 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.108987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.109756 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.109758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.110137 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.110138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.110291 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.110786 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:28.110796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:28.112192 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:28.113758 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.113765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.113766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.113963 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.114151 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.114151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.114152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.114235 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.114501 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:28.114505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:28.114610 (+ 105us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.114693 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.114803 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260812 14:54:28.115443 29612 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42449
I20260812 14:54:28.115532 29612 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:28.115690 29612 heartbeater.cc:511] Master 127.4.27.62:42449 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
I20260812 14:54:28.116199 29244 ts_manager.cc:194] Registered new tserver with Master: aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679)
I20260812 14:54:28.116794 29244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:41389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:54:28.130477 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42449
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:28.208442 29619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:28.208626 29619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:28.208649 29619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:28.209699 29619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:28.209733 29619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:28.209756 29619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:54:28.211165 29619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:28.211338 29619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:28.211503 29619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:28.216104 29634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.216125 29626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.216135 29627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.216166 29619 server_base.cc:1034] running on GCE node
I20260812 14:54:28.216498 29619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:28.216708 29619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:28.217839 29619 hybrid_clock.cc:648] HybridClock initialized: now 1786546468217827 us; error 31 us; skew 500 ppm
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:28.220153 29619 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:54:28.220515 29619 webserver.cc:460] Webserver started at http://127.4.27.2:45621/ using document root <none> and password file <none>
I20260812 14:54:28.220683 29619 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:28.220721 29619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:28.220808 29619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:28.221711 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "b7d795e8db554fff95a01a9c6fb4931a"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "877c79b4bc764389eb33b8f778c123bc"
server_key_iv: "6240466ea6d66fef453a99353bf776bd"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.221988 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "b7d795e8db554fff95a01a9c6fb4931a"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "877c79b4bc764389eb33b8f778c123bc"
server_key_iv: "6240466ea6d66fef453a99353bf776bd"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.223193 29619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:28.223811 29647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.223935 29619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:54:28.224012 29619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "b7d795e8db554fff95a01a9c6fb4931a"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "877c79b4bc764389eb33b8f778c123bc"
server_key_iv: "6240466ea6d66fef453a99353bf776bd"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.224076 29619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:28.234630 29619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:28.236660 29619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:28.236785 29619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:28.236994 29619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:28.237287 29619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:28.237330 29619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.237358 29619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:28.237385 29619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.242937 29619 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:35821
I20260812 14:54:28.242974 29859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:35821 every 8 connection(s)
I20260812 14:54:28.243258 29619 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:54:28.244726 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 29619
I20260812 14:54:28.244848 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:54:28.245051 4204 external_mini_cluster.cc:1428] Setting key ad56539e965c69a3c11992dd52eb0996
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:28.252353 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.243874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:56805 (local address 127.4.27.62:42449)
0812 14:54:28.243931 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.243933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.244103 (+ 170us) server_negotiation.cc:407] Connection header received
0812 14:54:28.244555 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.244556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.244569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.244618 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.245818 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.246318 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.247655 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.247944 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.248003 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.249885 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.249886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.249887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.249893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.251145 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.251494 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.251495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.251496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.251549 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.251762 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.251763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.251763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.251868 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.251924 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.252180 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.252266 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:54:28.252560 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.243857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:56805)
0812 14:54:28.243996 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.244029 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.244466 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.244704 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.244709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.245472 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:28.245717 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.245722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.246435 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.246436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.247562 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.247565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.248016 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.248016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.248110 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.248535 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:28.248543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:28.249800 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:28.251219 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.251225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.251227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.251428 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.251602 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.251603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.251603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.251706 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.251922 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:28.251929 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:28.252063 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.252151 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.252311 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260812 14:54:28.252988 29864 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42449
I20260812 14:54:28.253072 29864 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:28.253309 29864 heartbeater.cc:511] Master 127.4.27.62:42449 requested a full tablet report, sending...
I20260812 14:54:28.253697 29244 ts_manager.cc:194] Registered new tserver with Master: b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
I20260812 14:54:28.254190 29244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:56805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:54:28.268926 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42449
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:28.347604 29871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:28.347786 29871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:28.347808 29871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:28.348869 29871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:28.348902 29871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:28.348927 29871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:54:28.350288 29871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:28.350482 29871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:28.350662 29871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:28.355204 29881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.355198 29880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.355201 29886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.355254 29871 server_base.cc:1034] running on GCE node
I20260812 14:54:28.355515 29871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:28.355726 29871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:28.356858 29871 hybrid_clock.cc:648] HybridClock initialized: now 1786546468356848 us; error 31 us; skew 500 ppm
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:28.359172 29871 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:54:28.359534 29871 webserver.cc:460] Webserver started at http://127.4.27.3:40423/ using document root <none> and password file <none>
I20260812 14:54:28.359711 29871 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:28.359750 29871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:28.359836 29871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:28.360749 29871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "4899ce61dda54e858ecc37628336e945"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "615556fefd4fdf03dc0aa3a77a4ae17b"
server_key_iv: "23c4884dd9946c1fa70fcfcb2dc54103"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.361027 29871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "4899ce61dda54e858ecc37628336e945"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "615556fefd4fdf03dc0aa3a77a4ae17b"
server_key_iv: "23c4884dd9946c1fa70fcfcb2dc54103"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.362167 29871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:28.362915 29902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.363075 29871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.363159 29871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "4899ce61dda54e858ecc37628336e945"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "615556fefd4fdf03dc0aa3a77a4ae17b"
server_key_iv: "23c4884dd9946c1fa70fcfcb2dc54103"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.363211 29871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:28.382117 29871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:28.383944 29871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:28.384047 29871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:28.384227 29871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:28.384508 29871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:28.384546 29871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.384577 29871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:28.384599 29871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.390074 29871 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:45233
I20260812 14:54:28.390138 30123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:45233 every 8 connection(s)
I20260812 14:54:28.390447 29871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:54:28.393205 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 29871
I20260812 14:54:28.393323 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:54:28.393499 4204 external_mini_cluster.cc:1428] Setting key 4b7f7cd4d765f529f620898d5060cb51
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:28.398749 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.391069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:57905 (local address 127.4.27.62:42449)
0812 14:54:28.391119 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.391121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.391318 (+ 197us) server_negotiation.cc:407] Connection header received
0812 14:54:28.391760 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.391761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.391771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.391814 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.392648 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.393168 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.394462 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.394660 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.394711 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.396555 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.396556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.396557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.396563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.397756 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.398049 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.398050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.398050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.398096 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.398258 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.398258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.398259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.398374 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.398425 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.398627 (+ 202us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.398688 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:54:28.398896 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.391051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:57905)
0812 14:54:28.391212 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.391244 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.391688 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.391871 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.391874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.392392 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:28.392568 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.392573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.393247 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.393248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.394402 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.394405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.394716 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.394717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.394816 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.395188 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:28.395195 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:28.396479 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:28.397814 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.397817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.397818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.397998 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.398139 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.398140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.398140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.398217 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.398418 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:28.398422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:28.398529 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.398607 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.398719 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260812 14:54:28.399290 30126 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42449
I20260812 14:54:28.399365 30126 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:28.399492 30126 heartbeater.cc:511] Master 127.4.27.62:42449 requested a full tablet report, sending...
I20260812 14:54:28.399806 29244 ts_manager.cc:194] Registered new tserver with Master: 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233)
I20260812 14:54:28.400161 29244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:57905
I20260812 14:54:28.404940 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:54:28.411037 29354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.406133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.4.27.62:42449)
0812 14:54:28.406187 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.406189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.406256 (+ 67us) server_negotiation.cc:407] Connection header received
0812 14:54:28.406335 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.406336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.406364 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.406401 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.406886 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.407324 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.407980 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.408155 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.408205 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.409126 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.409127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.409128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.409134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.410050 (+ 916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.410361 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.410362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.410362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.410412 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.410569 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.410569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.410570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.410695 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.410788 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.410890 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.410960 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:54:28.412283 29244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:54:28.413739 29244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:54:28.419781 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.415995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35821 (local address 127.0.0.1:50346)
0812 14:54:28.416208 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.416220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.416278 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.416491 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.416492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.416495 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:28.416587 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.416589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.417564 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.417566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.418822 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.418825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.419492 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.419493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.419574 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.419595 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.419656 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.419708 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":7,"thread_start_us":42,"threads_started":1}
I20260812 14:54:28.419796 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.416177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.4.27.2:35821)
0812 14:54:28.416360 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.416364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.416373 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:54:28.416406 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.416407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.416418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.416491 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:28.416675 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.417472 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.418892 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.419407 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.419471 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.419528 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.419680 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.419717 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260812 14:54:28.420042 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.415807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37679 (local address 127.0.0.1:42660)
0812 14:54:28.416127 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.416168 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.416235 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.416482 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.416485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.416488 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:28.416553 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.416556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.417831 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.417832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.418918 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.418920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.419807 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.419808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.419875 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.419880 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.419929 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.419971 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":8,"thread_start_us":50,"threads_started":1}
I20260812 14:54:28.420163 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.416253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.4.27.3:45233)
0812 14:54:28.416404 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.416409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.416682 (+ 273us) server_negotiation.cc:407] Connection header received
0812 14:54:28.416721 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.416723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.416736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.416797 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:28.416964 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.417940 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.418962 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.419651 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.419714 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.419766 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.420047 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.420093 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260812 14:54:28.420186 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.416116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:45233 (local address 127.0.0.1:55592)
0812 14:54:28.416573 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.416582 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.416628 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.416861 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.416861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.416863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:28.416897 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.416899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.418022 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.418024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.418899 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.418901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.419802 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.419803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.419973 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.419976 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.420024 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.420055 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":43,"threads_started":1}
I20260812 14:54:28.420347 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.415938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.4.27.1:37679)
0812 14:54:28.416156 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.416161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.416322 (+ 161us) server_negotiation.cc:407] Connection header received
0812 14:54:28.416369 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.416371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.416386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.416464 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:28.416652 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.417728 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.418994 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.419647 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.419714 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.419766 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.420191 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.420274 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260812 14:54:28.420608 29741 tablet_service.cc:1467] Processing CreateTablet for tablet ee89419815ea4862b4f958157a619594 (DEFAULT_TABLE table=test-table [id=5ea84e7aef3940b89e7701e6615d3d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:28.420884 29981 tablet_service.cc:1467] Processing CreateTablet for tablet ee89419815ea4862b4f958157a619594 (DEFAULT_TABLE table=test-table [id=5ea84e7aef3940b89e7701e6615d3d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:28.420986 29741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee89419815ea4862b4f958157a619594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:28.420898 29490 tablet_service.cc:1467] Processing CreateTablet for tablet ee89419815ea4862b4f958157a619594 (DEFAULT_TABLE table=test-table [id=5ea84e7aef3940b89e7701e6615d3d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:28.421176 29490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee89419815ea4862b4f958157a619594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:28.421195 29981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee89419815ea4862b4f958157a619594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:28.423516 30158 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Bootstrap starting.
I20260812 14:54:28.424089 30158 tablet_bootstrap.cc:654] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:28.424181 30161 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Bootstrap starting.
I20260812 14:54:28.424629 30158 log.cc:824] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:28.424806 30161 tablet_bootstrap.cc:654] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:28.425295 30161 log.cc:824] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:28.425405 30158 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: No bootstrap required, opened a new log
I20260812 14:54:28.425473 30158 ts_tablet_manager.cc:1397] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:28.425639 30160 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Bootstrap starting.
I20260812 14:54:28.425968 30161 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: No bootstrap required, opened a new log
I20260812 14:54:28.426028 30161 ts_tablet_manager.cc:1397] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:28.426222 30160 tablet_bootstrap.cc:654] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:28.426779 30160 log.cc:824] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:28.427400 30158 raft_consensus.cc:348] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.427549 30160 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: No bootstrap required, opened a new log
I20260812 14:54:28.427569 30158 raft_consensus.cc:374] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:28.427601 30160 ts_tablet_manager.cc:1397] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:28.427611 30158 raft_consensus.cc:729] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa575bd50168442e99f224acaba4b49c, State: Initialized, Role: FOLLOWER
I20260812 14:54:28.427776 30158 consensus_queue.cc:260] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.427812 30161 raft_consensus.cc:348] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.427965 30161 raft_consensus.cc:374] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:28.428010 30161 raft_consensus.cc:729] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d795e8db554fff95a01a9c6fb4931a, State: Initialized, Role: FOLLOWER
I20260812 14:54:28.428083 30158 ts_tablet_manager.cc:1428] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:54:28.428123 30161 consensus_queue.cc:260] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.428236 29612 heartbeater.cc:503] Master 127.4.27.62:42449 was elected leader, sending a full tablet report...
I20260812 14:54:28.428382 30161 ts_tablet_manager.cc:1428] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:54:28.428419 29864 heartbeater.cc:503] Master 127.4.27.62:42449 was elected leader, sending a full tablet report...
I20260812 14:54:28.429422 30160 raft_consensus.cc:348] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.429570 30160 raft_consensus.cc:374] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:28.429602 30160 raft_consensus.cc:729] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4899ce61dda54e858ecc37628336e945, State: Initialized, Role: FOLLOWER
I20260812 14:54:28.429682 30160 consensus_queue.cc:260] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.429896 30160 ts_tablet_manager.cc:1428] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:54:28.429935 30126 heartbeater.cc:503] Master 127.4.27.62:42449 was elected leader, sending a full tablet report...
W20260812 14:54:28.493808 29865 tablet.cc:2369] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:54:28.511395 30171 raft_consensus.cc:482] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:28.511502 30171 raft_consensus.cc:504] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.511799 30171 leader_election.cc:288] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821), 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233)
I20260812 14:54:28.515363 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.512144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:52797 (local address 127.4.27.3:45233)
0812 14:54:28.512196 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.512198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.512318 (+ 120us) server_negotiation.cc:407] Connection header received
0812 14:54:28.512368 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.512369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.512382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.512432 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:28.512556 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.513247 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.514474 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.515020 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.515090 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.515100 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.515266 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.515302 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:54:28.515363 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.512060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:35435 (local address 127.4.27.2:35821)
0812 14:54:28.512107 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.512109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.512117 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:28.512142 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.512143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.512154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.512209 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:28.512372 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.513111 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.514455 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.515020 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.515090 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.515100 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.515256 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.515294 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 14:54:28.515383 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.511958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35821 (local address 127.4.27.1:35435)
0812 14:54:28.512016 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.512030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.512083 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.512225 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.512226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.512228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:28.512306 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.512308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.513184 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.513185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.514372 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.514375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.515099 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.515100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.515182 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.515186 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.515245 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.515290 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260812 14:54:28.515389 30178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.512062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:45233 (local address 127.4.27.1:52797)
0812 14:54:28.512249 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.512258 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.512317 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.512441 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.512441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.512442 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:28.512500 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.512501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.513308 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.513309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.514410 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.514412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.515099 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.515100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.515182 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.515186 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.515254 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.515288 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":13,"thread_start_us":112,"threads_started":1}
I20260812 14:54:28.515697 29778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee89419815ea4862b4f958157a619594" candidate_uuid: "aa575bd50168442e99f224acaba4b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d795e8db554fff95a01a9c6fb4931a" is_pre_election: true
I20260812 14:54:28.515780 30038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee89419815ea4862b4f958157a619594" candidate_uuid: "aa575bd50168442e99f224acaba4b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4899ce61dda54e858ecc37628336e945" is_pre_election: true
I20260812 14:54:28.515846 29778 raft_consensus.cc:2454] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa575bd50168442e99f224acaba4b49c in term 0.
I20260812 14:54:28.515940 30038 raft_consensus.cc:2454] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa575bd50168442e99f224acaba4b49c in term 0.
I20260812 14:54:28.516059 29396 leader_election.cc:302] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa575bd50168442e99f224acaba4b49c, b7d795e8db554fff95a01a9c6fb4931a; no voters:
I20260812 14:54:28.516192 30171 raft_consensus.cc:2781] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:54:28.516260 30171 raft_consensus.cc:482] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:54:28.516275 30171 raft_consensus.cc:3037] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:28.517031 30171 raft_consensus.cc:504] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.517151 30171 leader_election.cc:288] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 1 election: Requested vote from peers b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821), 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233)
I20260812 14:54:28.517550 29778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee89419815ea4862b4f958157a619594" candidate_uuid: "aa575bd50168442e99f224acaba4b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d795e8db554fff95a01a9c6fb4931a"
I20260812 14:54:28.517550 30038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee89419815ea4862b4f958157a619594" candidate_uuid: "aa575bd50168442e99f224acaba4b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4899ce61dda54e858ecc37628336e945"
I20260812 14:54:28.517717 30038 raft_consensus.cc:3037] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:28.517717 29778 raft_consensus.cc:3037] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:28.518858 30038 raft_consensus.cc:2454] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa575bd50168442e99f224acaba4b49c in term 1.
I20260812 14:54:28.518858 29778 raft_consensus.cc:2454] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa575bd50168442e99f224acaba4b49c in term 1.
I20260812 14:54:28.519045 29396 leader_election.cc:302] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4899ce61dda54e858ecc37628336e945, aa575bd50168442e99f224acaba4b49c; no voters:
I20260812 14:54:28.519134 30171 raft_consensus.cc:2781] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:28.519292 30171 raft_consensus.cc:686] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 LEADER]: Becoming Leader. State: Replica: aa575bd50168442e99f224acaba4b49c, State: Running, Role: LEADER
I20260812 14:54:28.519405 30171 consensus_queue.cc:237] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.520224 29244 catalog_manager.cc:5495] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c reported cstate change: term changed from 0 to 1, leader changed from <none> to aa575bd50168442e99f224acaba4b49c (127.4.27.1). New cstate: current_term: 1 leader_uuid: "aa575bd50168442e99f224acaba4b49c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } health_report { overall_health: UNKNOWN } } }
I20260812 14:54:28.537010 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.535037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.4.27.1:37679)
0812 14:54:28.535090 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.535091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.535099 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:28.535126 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.535126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.535143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.535203 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:54:28.535334 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.535767 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.536453 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.536623 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.536663 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.536765 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:54:28.536861 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:54:28.536915 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.536927 (+ 12us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.536943 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:54:28.538681 30038 raft_consensus.cc:1261] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Refusing update from remote peer aa575bd50168442e99f224acaba4b49c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:28.538682 29778 raft_consensus.cc:1261] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Refusing update from remote peer aa575bd50168442e99f224acaba4b49c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:28.539036 30180 consensus_queue.cc:1035] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:28.539170 30171 consensus_queue.cc:1035] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:28.542259 30191 mvcc.cc:205] Tried to move back new op lower bound from 7317694335131619328 to 7317694335055986688. Current Snapshot: MvccSnapshot[applied={T|T < 7317694335131619328 or (T in {7317694335131619328})}]
I20260812 14:54:28.543591 29241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:28.543672 29241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:28.544328 29241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Sending DeleteTablet for 3 replicas of tablet ee89419815ea4862b4f958157a619594
I20260812 14:54:28.544612 29490 tablet_service.cc:1514] Processing DeleteTablet for tablet ee89419815ea4862b4f958157a619594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:28 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42660
I20260812 14:54:28.544652 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 29364
I20260812 14:54:28.553086 29981 tablet_service.cc:1514] Processing DeleteTablet for tablet ee89419815ea4862b4f958157a619594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:28 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55592
I20260812 14:54:28.553445 29741 tablet_service.cc:1514] Processing DeleteTablet for tablet ee89419815ea4862b4f958157a619594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:28 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:50346
I20260812 14:54:28.553699 30203 tablet_replica.cc:330] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: stopping tablet replica
I20260812 14:54:28.553830 30203 raft_consensus.cc:2229] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:54:28.553831 30202 tablet_replica.cc:330] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: stopping tablet replica
I20260812 14:54:28.553933 30202 raft_consensus.cc:2229] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:54:28.553951 30203 raft_consensus.cc:2258] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:28.554044 30202 raft_consensus.cc:2258] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:28.554409 30203 ts_tablet_manager.cc:1905] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:28.554711 30202 ts_tablet_manager.cc:1905] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:28.555779 30203 ts_tablet_manager.cc:1918] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:28.555846 30203 log.cc:1196] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/ee89419815ea4862b4f958157a619594
I20260812 14:54:28.556087 30203 ts_tablet_manager.cc:1939] T ee89419815ea4862b4f958157a619594 P b7d795e8db554fff95a01a9c6fb4931a: Deleting consensus metadata
I20260812 14:54:28.556460 29217 catalog_manager.cc:4841] TS b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): tablet ee89419815ea4862b4f958157a619594 (table test-table [id=5ea84e7aef3940b89e7701e6615d3d48]) successfully deleted
I20260812 14:54:28.556624 30202 ts_tablet_manager.cc:1918] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:28.556715 30202 log.cc:1196] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/ee89419815ea4862b4f958157a619594
I20260812 14:54:28.557053 30202 ts_tablet_manager.cc:1939] T ee89419815ea4862b4f958157a619594 P 4899ce61dda54e858ecc37628336e945: Deleting consensus metadata
I20260812 14:54:28.557440 29217 catalog_manager.cc:4841] TS 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): tablet ee89419815ea4862b4f958157a619594 (table test-table [id=5ea84e7aef3940b89e7701e6615d3d48]) successfully deleted
W20260812 14:54:28.561209 29217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:37679 (error 108)
I20260812 14:54:28.561319 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 29619
I20260812 14:54:28.563079 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.562952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37679 (local address 127.0.0.1:42678)
0812 14:54:28.563007 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.563049 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:37679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:54:28.563195 29217 catalog_manager.cc:4584] TS aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee89419815ea4862b4f958157a619594: Network error: Client connection negotiation failed: client connection to 127.4.27.1:37679: connect: Connection refused (error 111)
I20260812 14:54:28.567577 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 29871
I20260812 14:54:28.579348 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 29177
I20260812 14:54:28.591343 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42449
--webserver_interface=127.4.27.62
--webserver_port=35257
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:28.673722 30205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:28.673936 30205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:28.673959 30205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:28.675292 30205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:28.675338 30205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:28.675351 30205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:28.675362 30205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:28.675374 30205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:28.676827 30205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:28.677042 30205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:28.677248 30205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:28.685982 30213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.686020 30205 server_base.cc:1034] running on GCE node
W20260812 14:54:28.686077 30212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.685981 30218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.686455 30205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:28.686759 30205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:28.687899 30205 hybrid_clock.cc:648] HybridClock initialized: now 1786546468687886 us; error 27 us; skew 500 ppm
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:28.690263 30205 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:28.690704 30205 webserver.cc:460] Webserver started at http://127.4.27.62:35257/ using document root <none> and password file <none>
I20260812 14:54:28.690912 30205 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:28.690975 30205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:28.692353 30205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:28.692967 30237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.693130 30205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:54:28.693238 30205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "40a77d4c5e9d4d50bb148720dceff5bd"
format_stamp: "Formatted at 2026-08-12 14:54:27 on dist-test-slave-6zbq"
server_key: "8b27bb6ee5e989f73dce08758807ffa6"
server_key_iv: "1438787139a603a8e2e383784a273774"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.693630 30205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:28.707974 30205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:28.709726 30205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:28.709867 30205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:28.713629 30205 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:42449
I20260812 14:54:28.713657 30342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:42449 every 8 connection(s)
I20260812 14:54:28.713999 30205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:28.715582 30343 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:54:28.716286 30343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap starting.
I20260812 14:54:28.716310 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 30205
I20260812 14:54:28.716607 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:37679
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:42449
--webserver_port=35417
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:28.724870 30343 log.cc:824] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:28.726765 30343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:54:28.726997 30343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap complete.
I20260812 14:54:28.729058 30343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:28.729277 30343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a77d4c5e9d4d50bb148720dceff5bd, State: Initialized, Role: FOLLOWER
I20260812 14:54:28.729411 30343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:28.729468 30343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:28.729504 30343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:28.729549 30343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:54:28.730518 30343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:28.730605 30343 leader_election.cc:302] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a77d4c5e9d4d50bb148720dceff5bd; no voters:
I20260812 14:54:28.730746 30343 leader_election.cc:288] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:54:28.730800 30351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:54:28.730967 30351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 LEADER]: Becoming Leader. State: Replica: 40a77d4c5e9d4d50bb148720dceff5bd, State: Running, Role: LEADER
I20260812 14:54:28.730998 30343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:28.731112 30351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:28.731374 30351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a77d4c5e9d4d50bb148720dceff5bd. Latest consensus state: current_term: 2 leader_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } } }
I20260812 14:54:28.731396 30355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } } }
I20260812 14:54:28.731536 30355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:28.731881 30364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:28.731891 30351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:28.732615 30364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:28.732775 30364 catalog_manager.cc:1259] Loaded cluster ID: 86c1dacef716448bad683e2b373a135d
I20260812 14:54:28.732960 30364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:54:28.733217 30364 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 14:54:28.737875 30364 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 14:54:28.740130 30364 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:28.740316 30364 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Loaded TSK: 0
I20260812 14:54:28.740521 30364 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:54:28.798255 30346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:28.798456 30346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:28.798480 30346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:28.799597 30346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:28.799643 30346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:28.799669 30346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:54:28.801375 30346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:37679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=35417
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:28.801604 30346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:28.801826 30346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:28.806582 30389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.806582 30388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.806645 30393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.806871 30346 server_base.cc:1034] running on GCE node
I20260812 14:54:28.807034 30346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:28.807245 30346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:28.808410 30346 hybrid_clock.cc:648] HybridClock initialized: now 1786546468808395 us; error 33 us; skew 500 ppm
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:28.811254 30346 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:54:28.811794 30346 webserver.cc:460] Webserver started at http://127.4.27.1:35417/ using document root <none> and password file <none>
I20260812 14:54:28.812063 30346 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:28.812143 30346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:28.813844 30346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:54:28.814600 30408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.814756 30346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.814859 30346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "aa575bd50168442e99f224acaba4b49c"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "62a20c86b186ec3a40aefc2c2d3eac98"
server_key_iv: "6d69a47af94a8135cfbfc01509b48fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.815292 30346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:28.835779 30346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:28.837815 30346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:28.837944 30346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:28.838141 30346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:28.838626 30425 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:54:28.840621 30346 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:54:28.840677 30346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:54:28.840709 30346 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:54:28.841351 30346 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:54:28.841382 30346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:28.841481 30425 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Bootstrap starting.
I20260812 14:54:28.848621 30346 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37679
I20260812 14:54:28.848788 30640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37679 every 8 connection(s)
I20260812 14:54:28.849057 30346 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:54:28.851723 30425 log.cc:824] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:28.852586 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 30346
I20260812 14:54:28.852878 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:35821
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:42449
--webserver_port=45621
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:28.854393 30425 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:54:28.854665 30425 tablet_bootstrap.cc:492] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Bootstrap complete.
I20260812 14:54:28.855695 30425 ts_tablet_manager.cc:1397] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:28.861254 30425 raft_consensus.cc:348] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.861596 30425 raft_consensus.cc:729] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa575bd50168442e99f224acaba4b49c, State: Initialized, Role: FOLLOWER
I20260812 14:54:28.861739 30425 consensus_queue.cc:260] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } }
I20260812 14:54:28.862000 30425 ts_tablet_manager.cc:1428] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260812 14:54:28.862767 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.849778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:44573)
0812 14:54:28.850033 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.850074 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.850601 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.851004 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.851008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.851284 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:28.851464 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.851469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.852968 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.852969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.854543 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.854546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.855460 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.855462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.855655 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.856327 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:28.856337 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:28.858324 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:28.860869 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.860877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.860879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.861193 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.861529 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.861530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.861530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.861669 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.862137 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:28.862142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:28.862291 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.862407 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.862568 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260812 14:54:28.862869 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.849862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:44573 (local address 127.4.27.62:42449)
0812 14:54:28.850039 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.850048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.850157 (+ 109us) server_negotiation.cc:407] Connection header received
0812 14:54:28.850808 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.850813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.850857 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.850969 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.851578 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.852830 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.854708 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.855342 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.855446 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.858455 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.858459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.858460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.858473 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.860769 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.861306 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.861309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.861310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.861448 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.861753 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.861754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.861754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.862036 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.862109 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.862518 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.862669 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260812 14:54:28.864298 30641 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42449
I20260812 14:54:28.864396 30641 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:28.864560 30641 heartbeater.cc:511] Master 127.4.27.62:42449 requested a full tablet report, sending...
I20260812 14:54:28.865339 30266 ts_manager.cc:194] Registered new tserver with Master: aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679)
I20260812 14:54:28.866534 30266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:44573
I20260812 14:54:28.867530 30641 heartbeater.cc:503] Master 127.4.27.62:42449 was elected leader, sending a full tablet report...
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.1@KRBTEST.COM
I20260812 14:54:28.874971 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.865960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37679 (local address 127.0.0.1:42682)
0812 14:54:28.866449 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.866483 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.866546 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.867811 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.867818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.868109 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:28.868162 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.868164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.869302 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.869303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.870157 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.870160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.870646 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.870650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.870720 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.871137 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:28.871141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:28.872330 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:28.873852 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.873859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.873861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.874050 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.874243 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.874243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.874244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.874306 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.874608 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:28.874615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:28.874669 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.874743 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.874876 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":90,"threads_started":1}
I20260812 14:54:28.874972 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.866865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.4.27.1:37679)
0812 14:54:28.867003 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.867008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.867017 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:54:28.867047 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.867049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.867065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.868240 (+ 1175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.868309 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.869194 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.870235 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.870578 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.870640 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.872412 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.872420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.872420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.872427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.873759 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.874127 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.874130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.874130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.874185 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.874413 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.874414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.874414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.874542 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.874615 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.874794 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.874874 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260812 14:54:28.875351 30508 tablet_service.cc:1514] Processing DeleteTablet for tablet ee89419815ea4862b4f958157a619594 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:28 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42682
I20260812 14:54:28.875831 30651 tablet_replica.cc:330] stopping tablet replica
I20260812 14:54:28.875927 30651 raft_consensus.cc:2229] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:54:28.875998 30651 raft_consensus.cc:2258] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:28.876281 30651 ts_tablet_manager.cc:1905] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:28.878098 30651 ts_tablet_manager.cc:1918] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:28.878163 30651 log.cc:1196] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/ee89419815ea4862b4f958157a619594
I20260812 14:54:28.878391 30651 ts_tablet_manager.cc:1939] T ee89419815ea4862b4f958157a619594 P aa575bd50168442e99f224acaba4b49c: Deleting consensus metadata
I20260812 14:54:28.878866 30241 catalog_manager.cc:4841] TS aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679): tablet ee89419815ea4862b4f958157a619594 (table test-table [id=5ea84e7aef3940b89e7701e6615d3d48]) successfully deleted
W20260812 14:54:28.941445 30647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:28.941630 30647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:28.941660 30647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:28.942818 30647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:28.942858 30647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:28.942893 30647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:54:28.944309 30647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:35821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=45621
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:28.944521 30647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:28.944726 30647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:28.949321 30673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.949337 30656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:28.949337 30657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:28.949568 30647 server_base.cc:1034] running on GCE node
I20260812 14:54:28.949762 30647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:28.949955 30647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:28.951082 30647 hybrid_clock.cc:648] HybridClock initialized: now 1786546468951067 us; error 28 us; skew 500 ppm
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:28.953372 30647 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:54:28.953761 30647 webserver.cc:460] Webserver started at http://127.4.27.2:45621/ using document root <none> and password file <none>
I20260812 14:54:28.953969 30647 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:28.954030 30647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:28.955287 30647 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:28.955770 30687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.955915 30647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.955996 30647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "b7d795e8db554fff95a01a9c6fb4931a"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "877c79b4bc764389eb33b8f778c123bc"
server_key_iv: "6240466ea6d66fef453a99353bf776bd"
server_key_version: "encryptionkey@0"
I20260812 14:54:28.956300 30647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:28.972910 30647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:28.974711 30647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:28.974836 30647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:28.975026 30647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:28.975294 30647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:28.975327 30647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.975353 30647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:28.975370 30647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:28.980592 30647 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:35821
I20260812 14:54:28.980628 30891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:35821 every 8 connection(s)
I20260812 14:54:28.980921 30647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:54:28.982564 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 30647
I20260812 14:54:28.982889 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:45233
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:42449
--webserver_port=40423
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 14:54:28 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:28.990259 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.981569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:52025 (local address 127.4.27.62:42449)
0812 14:54:28.981613 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:28.981615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:28.981797 (+ 182us) server_negotiation.cc:407] Connection header received
0812 14:54:28.982218 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:28.982219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:28.982227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:28.982263 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:28.983366 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.983842 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.984991 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.985281 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.985358 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.987729 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:28.987731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:28.987731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:28.987738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:28.988954 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.989402 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.989403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.989403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.989448 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:28.989694 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:28.989694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:28.989695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:28.989785 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:28.989829 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:28.990131 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 14:54:28.990197 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:54:28.990523 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:28.981544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:52025)
0812 14:54:28.981686 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:28.981724 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:28.982134 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:28.982318 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:28.982321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:28.983023 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:28.983272 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.983278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.983935 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.983936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.984900 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:28.984902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:28.985393 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:28.985394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:28.985509 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:28.985931 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:28.985939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:28.987635 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:28.989032 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.989040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.989042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.989330 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.989501 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:28.989502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:28.989502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:28.989636 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:28.989839 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:28.989845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:28.990004 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 14:54:28.990120 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 14:54:28.990277 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260812 14:54:28.991015 30893 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42449
I20260812 14:54:28.991097 30893 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:28.991284 30893 heartbeater.cc:511] Master 127.4.27.62:42449 requested a full tablet report, sending...
I20260812 14:54:28.991631 30266 ts_manager.cc:194] Registered new tserver with Master: b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
I20260812 14:54:28.991968 30266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:52025
W20260812 14:54:29.064611 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:29.064797 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:29.064827 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:29.065917 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:29.065958 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:29.065992 30900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:54:29.067412 30900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:45233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=40423
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:29.067627 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:29.067821 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:29.072404 30911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.072399 30900 server_base.cc:1034] running on GCE node
W20260812 14:54:29.072384 30914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.072377 30912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.072829 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:29.073051 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:29.074187 30900 hybrid_clock.cc:648] HybridClock initialized: now 1786546469074175 us; error 24 us; skew 500 ppm
Aug 12 14:54:29 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:29.076531 30900 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:54:29.076927 30900 webserver.cc:460] Webserver started at http://127.4.27.3:40423/ using document root <none> and password file <none>
I20260812 14:54:29.077091 30900 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:29.077139 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:29.078301 30900 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:29.078812 30930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.078960 30900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.079046 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "4899ce61dda54e858ecc37628336e945"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "615556fefd4fdf03dc0aa3a77a4ae17b"
server_key_iv: "23c4884dd9946c1fa70fcfcb2dc54103"
server_key_version: "encryptionkey@0"
I20260812 14:54:29.079326 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:29.099143 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:29.101150 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:29.101255 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:29.101436 30900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:29.101671 30900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:29.101701 30900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.101720 30900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:29.101732 30900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.107089 30900 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:45233
I20260812 14:54:29.107141 31164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:45233 every 8 connection(s)
I20260812 14:54:29.107437 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:54:29.108496 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 30900
Aug 12 14:54:29 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:29.115298 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.108845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.4.27.62:42449)
0812 14:54:29.108937 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.108938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.108991 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:54:29.109062 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.109062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.109070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.109118 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:29.109789 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.110543 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.111554 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.111750 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.111792 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.112872 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:29.112874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:29.112874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:29.112881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:29.114149 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:29.114509 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:29.114510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:29.114510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:29.114568 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:29.114749 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:29.114749 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:29.114750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:29.114881 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:29.115038 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.115080 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.115198 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":56,"threads_started":1}
I20260812 14:54:29.116816 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.108044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:55867 (local address 127.4.27.62:42449)
0812 14:54:29.108089 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.108091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.108288 (+ 197us) server_negotiation.cc:407] Connection header received
0812 14:54:29.108763 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.108764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.108777 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.108843 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:29.109959 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.110634 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.111861 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.112027 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.112073 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.114159 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:29.114161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:29.114161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:29.114167 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:29.115592 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:29.115972 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:29.115974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:29.115974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:29.116036 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:29.116247 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:29.116248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:29.116249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:29.116358 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:29.116413 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.116660 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.116737 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:54:29.116974 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.108034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:55867)
0812 14:54:29.108179 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.108212 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:29.108666 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:29.108920 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:29.108924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:29.109621 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:29.109878 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.109883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.110718 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.110719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.111781 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.111785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.112102 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.112103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.112243 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.112646 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:29.112653 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:29.114073 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:29.115656 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:29.115663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:29.115665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:29.115904 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:29.116085 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:29.116086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:29.116086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:29.116191 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:29.116413 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:29.116419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:29.116545 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 14:54:29.116628 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:54:29.116780 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260812 14:54:29.117354 31166 heartbeater.cc:347] Connected to a master server at 127.4.27.62:42449
I20260812 14:54:29.117442 31166 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:29.117599 31166 heartbeater.cc:511] Master 127.4.27.62:42449 requested a full tablet report, sending...
I20260812 14:54:29.117911 30266 ts_manager.cc:194] Registered new tserver with Master: 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233)
I20260812 14:54:29.118240 30266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:55867
I20260812 14:54:29.126794 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:54:29.129081 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.127098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.4.27.62:42449)
0812 14:54:29.127159 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.127161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.127223 (+ 62us) server_negotiation.cc:407] Connection header received
0812 14:54:29.127299 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.127300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.127332 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.127377 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:54:29.127455 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.127893 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.128552 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.128716 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.128760 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.128867 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:54:29.128948 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:54:29.128999 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.129010 (+ 11us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.129022 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:54:29.129343 30266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:54:29.130954 30266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:54:29.133126 30508 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb5f603c4bb468eb691763dfcd0aa20 (DEFAULT_TABLE table=test-table [id=f3fca2fb3e5447018069baac2c1a703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:29.133355 30508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb5f603c4bb468eb691763dfcd0aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:29.134984 30425 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Bootstrap starting.
I20260812 14:54:29.135316 30425 tablet_bootstrap.cc:654] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:29.135792 30425 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: No bootstrap required, opened a new log
I20260812 14:54:29.135836 30425 ts_tablet_manager.cc:1397] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:29.136133 30425 raft_consensus.cc:348] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.136214 30425 raft_consensus.cc:374] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:29.136281 30425 raft_consensus.cc:729] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa575bd50168442e99f224acaba4b49c, State: Initialized, Role: FOLLOWER
I20260812 14:54:29.136329 30425 consensus_queue.cc:260] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.136415 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.132685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35821 (local address 127.0.0.1:50358)
0812 14:54:29.132846 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.132854 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:29.132897 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:29.133214 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:29.133215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:29.133218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:29.133262 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.133264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.134189 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.134190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.135200 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.135203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.136157 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.136158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.136220 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.136253 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 14:54:29.136309 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:54:29.136349 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260812 14:54:29.136425 30425 ts_tablet_manager.cc:1428] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 14:54:29.136576 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.132599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:45233 (local address 127.0.0.1:55596)
0812 14:54:29.132641 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.132651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:29.132686 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:29.133224 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:29.133225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:29.133227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:29.133284 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.133285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.134544 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.134545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.135512 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.135516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.136361 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.136363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.136433 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.136437 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:29.136465 (+ 28us) client_negotiation.cc:240] Negotiation successful
0812 14:54:29.136510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:54:29.136694 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.132874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.4.27.2:35821)
0812 14:54:29.133043 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.133047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.133059 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:54:29.133094 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.133096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.133108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.133190 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:29.133387 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.134080 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.135300 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.136058 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.136125 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.136182 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.136570 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.136617 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260812 14:54:29.136929 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.132742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.4.27.3:45233)
0812 14:54:29.133048 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.133052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.133062 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:54:29.133096 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.133098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.133112 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.133356 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:29.133418 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.134427 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.135610 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.136259 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.136728 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.136798 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.136822 (+ 24us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.136857 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":101,"threads_started":1}
I20260812 14:54:29.137420 30771 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb5f603c4bb468eb691763dfcd0aa20 (DEFAULT_TABLE table=test-table [id=f3fca2fb3e5447018069baac2c1a703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:29.137503 31017 tablet_service.cc:1467] Processing CreateTablet for tablet 6bb5f603c4bb468eb691763dfcd0aa20 (DEFAULT_TABLE table=test-table [id=f3fca2fb3e5447018069baac2c1a703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:29.137722 30771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb5f603c4bb468eb691763dfcd0aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:29.137811 31017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bb5f603c4bb468eb691763dfcd0aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:29.140365 31183 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Bootstrap starting.
I20260812 14:54:29.140422 31185 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Bootstrap starting.
I20260812 14:54:29.140951 31183 tablet_bootstrap.cc:654] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:29.140951 31185 tablet_bootstrap.cc:654] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:29.141387 31183 log.cc:824] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:29.141388 31185 log.cc:824] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:29.141987 31183 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: No bootstrap required, opened a new log
I20260812 14:54:29.142015 31185 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: No bootstrap required, opened a new log
I20260812 14:54:29.142040 31183 ts_tablet_manager.cc:1397] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:29.142071 31185 ts_tablet_manager.cc:1397] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:29.143452 31185 raft_consensus.cc:348] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.143478 31183 raft_consensus.cc:348] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.143590 31183 raft_consensus.cc:374] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:29.143585 31185 raft_consensus.cc:374] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:29.143611 31183 raft_consensus.cc:729] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d795e8db554fff95a01a9c6fb4931a, State: Initialized, Role: FOLLOWER
I20260812 14:54:29.143618 31185 raft_consensus.cc:729] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4899ce61dda54e858ecc37628336e945, State: Initialized, Role: FOLLOWER
I20260812 14:54:29.143711 31183 consensus_queue.cc:260] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.143759 31185 consensus_queue.cc:260] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.143959 31183 ts_tablet_manager.cc:1428] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:54:29.143987 30893 heartbeater.cc:503] Master 127.4.27.62:42449 was elected leader, sending a full tablet report...
I20260812 14:54:29.144024 31185 ts_tablet_manager.cc:1428] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:54:29.144049 31166 heartbeater.cc:503] Master 127.4.27.62:42449 was elected leader, sending a full tablet report...
W20260812 14:54:29.231431 30895 tablet.cc:2369] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:54:29.305311 31193 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:29.305437 31193 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.305840 31193 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821), aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679)
I20260812 14:54:29.308507 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.306024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35821 (local address 127.4.27.3:43555)
0812 14:54:29.306079 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.306091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:29.306144 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:29.306300 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:29.306301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:29.306303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:29.306399 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.306400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.307010 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.307011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.307728 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.307730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.308262 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.308263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.308348 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.308351 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:54:29.308397 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:54:29.308436 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:54:29.308507 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.306101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:43555 (local address 127.4.27.2:35821)
0812 14:54:29.306164 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.306167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.306179 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:54:29.306208 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.306209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.306220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.306290 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:29.306465 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.306934 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.307819 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.308184 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.308242 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.308254 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.308409 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.308446 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:54:29.308825 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.306211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:58749 (local address 127.4.27.1:37679)
0812 14:54:29.306270 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.306273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.306389 (+ 116us) server_negotiation.cc:407] Connection header received
0812 14:54:29.306448 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.306449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.306477 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.306531 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:29.306698 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.307195 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.308094 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.308498 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.308559 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.308635 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.308724 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.308765 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:54:29.308827 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.306128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37679 (local address 127.4.27.3:58749)
0812 14:54:29.306319 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.306327 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:29.306394 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:29.306544 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:29.306545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:29.306547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:29.306632 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.306634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.307274 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.307275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.308005 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:29.308007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.308580 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.308582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:29.308666 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.308670 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:29.308718 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 14:54:29.308752 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":124,"threads_started":1}
I20260812 14:54:29.308848 30806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb5f603c4bb468eb691763dfcd0aa20" candidate_uuid: "4899ce61dda54e858ecc37628336e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d795e8db554fff95a01a9c6fb4931a" is_pre_election: true
I20260812 14:54:29.308986 30806 raft_consensus.cc:2454] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4899ce61dda54e858ecc37628336e945 in term 0.
I20260812 14:54:29.309219 30935 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4899ce61dda54e858ecc37628336e945, b7d795e8db554fff95a01a9c6fb4931a; no voters:
I20260812 14:54:29.309304 30545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb5f603c4bb468eb691763dfcd0aa20" candidate_uuid: "4899ce61dda54e858ecc37628336e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa575bd50168442e99f224acaba4b49c" is_pre_election: true
I20260812 14:54:29.309367 31193 raft_consensus.cc:2781] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:54:29.309424 31193 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:54:29.309454 31193 raft_consensus.cc:3037] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:29.309455 30545 raft_consensus.cc:2454] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4899ce61dda54e858ecc37628336e945 in term 0.
I20260812 14:54:29.310135 31193 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.310241 31193 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [CANDIDATE]: Term 1 election: Requested vote from peers b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821), aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679)
I20260812 14:54:29.310428 30806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb5f603c4bb468eb691763dfcd0aa20" candidate_uuid: "4899ce61dda54e858ecc37628336e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d795e8db554fff95a01a9c6fb4931a"
I20260812 14:54:29.310467 30545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bb5f603c4bb468eb691763dfcd0aa20" candidate_uuid: "4899ce61dda54e858ecc37628336e945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa575bd50168442e99f224acaba4b49c"
I20260812 14:54:29.310508 30806 raft_consensus.cc:3037] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:29.310530 30545 raft_consensus.cc:3037] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:29.311117 30545 raft_consensus.cc:2454] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4899ce61dda54e858ecc37628336e945 in term 1.
I20260812 14:54:29.311211 30806 raft_consensus.cc:2454] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4899ce61dda54e858ecc37628336e945 in term 1.
I20260812 14:54:29.311271 30935 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4899ce61dda54e858ecc37628336e945, aa575bd50168442e99f224acaba4b49c; no voters:
I20260812 14:54:29.311367 31193 raft_consensus.cc:2781] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:29.311501 31193 raft_consensus.cc:686] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 1 LEADER]: Becoming Leader. State: Replica: 4899ce61dda54e858ecc37628336e945, State: Running, Role: LEADER
I20260812 14:54:29.311578 31193 consensus_queue.cc:237] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.312418 30266 catalog_manager.cc:5495] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4899ce61dda54e858ecc37628336e945 (127.4.27.3). New cstate: current_term: 1 leader_uuid: "4899ce61dda54e858ecc37628336e945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } health_report { overall_health: UNKNOWN } } }
W20260812 14:54:29.349745 30642 tablet.cc:2369] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:54:29.357929 31167 tablet.cc:2369] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:54:29.361513 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.359529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.4.27.3:45233)
0812 14:54:29.359584 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:29.359585 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:29.359593 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:29.359623 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:29.359624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:29.359642 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:29.359698 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:54:29.359796 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.360241 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.360942 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:29.361127 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:29.361170 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:29.361272 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:54:29.361362 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:54:29.361416 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:29.361429 (+ 13us) server_negotiation.cc:299] Negotiation successful
0812 14:54:29.361445 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:54:29.363107 30806 raft_consensus.cc:1261] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Refusing update from remote peer 4899ce61dda54e858ecc37628336e945: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:29.363106 30545 raft_consensus.cc:1261] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Refusing update from remote peer 4899ce61dda54e858ecc37628336e945: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:29.363440 31193 consensus_queue.cc:1035] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:29.363580 31196 consensus_queue.cc:1035] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:29.365744 31203 mvcc.cc:205] Tried to move back new op lower bound from 7317694338508914688 to 7317694338300870656. Current Snapshot: MvccSnapshot[applied={T|T < 7317694338508914688}]
I20260812 14:54:29.365855 31204 mvcc.cc:205] Tried to move back new op lower bound from 7317694338508914688 to 7317694338300870656. Current Snapshot: MvccSnapshot[applied={T|T < 7317694338508914688}]
I20260812 14:54:29.366647 31205 mvcc.cc:205] Tried to move back new op lower bound from 7317694338508914688 to 7317694338300870656. Current Snapshot: MvccSnapshot[applied={T|T < 7317694338508914688}]
I20260812 14:54:29.368088 30266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:29.368176 30266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:29.368870 30266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Sending DeleteTablet for 3 replicas of tablet 6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:29.369148 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 30346
I20260812 14:54:29.369138 31017 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb5f603c4bb468eb691763dfcd0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:29 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:55596
I20260812 14:54:29.369330 30508 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb5f603c4bb468eb691763dfcd0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:29 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:42682
I20260812 14:54:29.369632 30771 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bb5f603c4bb468eb691763dfcd0aa20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:29 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:50358
I20260812 14:54:29.369771 31217 tablet_replica.cc:330] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: stopping tablet replica
I20260812 14:54:29.369869 31217 raft_consensus.cc:2229] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:54:29.369894 31218 tablet_replica.cc:330] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: stopping tablet replica
I20260812 14:54:29.369963 31217 raft_consensus.cc:2258] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:29.370105 31218 raft_consensus.cc:2229] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:54:29.370272 31218 raft_consensus.cc:2258] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:29.375082 31217 ts_tablet_manager.cc:1905] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:29.376147 31218 ts_tablet_manager.cc:1905] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:29.376721 31217 ts_tablet_manager.cc:1918] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:29.376798 31217 log.cc:1196] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:29.377075 31217 ts_tablet_manager.cc:1939] T 6bb5f603c4bb468eb691763dfcd0aa20 P 4899ce61dda54e858ecc37628336e945: Deleting consensus metadata
I20260812 14:54:29.377457 30241 catalog_manager.cc:4841] TS 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): tablet 6bb5f603c4bb468eb691763dfcd0aa20 (table test-table [id=f3fca2fb3e5447018069baac2c1a703c]) successfully deleted
I20260812 14:54:29.378378 31218 ts_tablet_manager.cc:1918] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:29.378556 31218 log.cc:1196] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:29.378949 31218 ts_tablet_manager.cc:1939] T 6bb5f603c4bb468eb691763dfcd0aa20 P b7d795e8db554fff95a01a9c6fb4931a: Deleting consensus metadata
I20260812 14:54:29.382190 30241 catalog_manager.cc:4841] TS b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): tablet 6bb5f603c4bb468eb691763dfcd0aa20 (table test-table [id=f3fca2fb3e5447018069baac2c1a703c]) successfully deleted
W20260812 14:54:29.387521 30241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:37679 (error 108)
I20260812 14:54:29.387926 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 30647
I20260812 14:54:29.388013 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.387883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37679 (local address 127.0.0.1:42688)
0812 14:54:29.387941 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.387983 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:37679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260812 14:54:29.398376 30241 catalog_manager.cc:4584] TS aa575bd50168442e99f224acaba4b49c (127.4.27.1:37679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bb5f603c4bb468eb691763dfcd0aa20: Network error: Client connection negotiation failed: client connection to 127.4.27.1:37679: connect: Connection refused (error 111)
I20260812 14:54:29.401768 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 30900
I20260812 14:54:29.414215 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 30205
I20260812 14:54:29.427242 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42449
--webserver_interface=127.4.27.62
--webserver_port=35257
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:29.506393 31224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:29.506574 31224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:29.506603 31224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:29.507688 31224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:29.507735 31224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:29.507754 31224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:29.507769 31224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:29.507786 31224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:29.509245 31224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:29.509477 31224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:29.509673 31224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:29.514286 31240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.514318 31236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.514324 31238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.514382 31224 server_base.cc:1034] running on GCE node
I20260812 14:54:29.514762 31224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:29.514987 31224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:29.516129 31224 hybrid_clock.cc:648] HybridClock initialized: now 1786546469516117 us; error 27 us; skew 500 ppm
Aug 12 14:54:29 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:29.523686 31224 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:29.524073 31224 webserver.cc:460] Webserver started at http://127.4.27.62:35257/ using document root <none> and password file <none>
I20260812 14:54:29.524296 31224 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:29.524346 31224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:29.525525 31224 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:29.526083 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.526228 31224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.526324 31224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "40a77d4c5e9d4d50bb148720dceff5bd"
format_stamp: "Formatted at 2026-08-12 14:54:27 on dist-test-slave-6zbq"
server_key: "8b27bb6ee5e989f73dce08758807ffa6"
server_key_iv: "1438787139a603a8e2e383784a273774"
server_key_version: "encryptionkey@0"
I20260812 14:54:29.526656 31224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:29.537868 31224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:29.539417 31224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:29.539541 31224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:29.543195 31224 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:42449
I20260812 14:54:29.543226 31358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:42449 every 8 connection(s)
I20260812 14:54:29.543545 31224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:29.544914 31361 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:54:29.545521 31361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap starting.
I20260812 14:54:29.552027 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 31224
I20260812 14:54:29.552392 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:37679
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:42449
--webserver_port=35417
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:29.553925 31361 log.cc:824] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:29.556113 31361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:54:29.556366 31361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd: Bootstrap complete.
I20260812 14:54:29.558393 31361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:29.558636 31361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a77d4c5e9d4d50bb148720dceff5bd, State: Initialized, Role: FOLLOWER
I20260812 14:54:29.558781 31361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:29.558852 31361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:29.558894 31361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:29.558939 31361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:54:29.559839 31361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:29.559922 31361 leader_election.cc:302] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a77d4c5e9d4d50bb148720dceff5bd; no voters:
I20260812 14:54:29.560076 31361 leader_election.cc:288] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 14:54:29.560123 31367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:54:29.560243 31367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [term 3 LEADER]: Becoming Leader. State: Replica: 40a77d4c5e9d4d50bb148720dceff5bd, State: Running, Role: LEADER
I20260812 14:54:29.560320 31361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:29.560336 31367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } }
I20260812 14:54:29.560654 31371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } } }
I20260812 14:54:29.560887 31371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:29.560654 31370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a77d4c5e9d4d50bb148720dceff5bd. Latest consensus state: current_term: 3 leader_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a77d4c5e9d4d50bb148720dceff5bd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42449 } } }
I20260812 14:54:29.561172 31370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a77d4c5e9d4d50bb148720dceff5bd [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:29.561259 31388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:29.562000 31388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:29.562103 31388 catalog_manager.cc:1259] Loaded cluster ID: 86c1dacef716448bad683e2b373a135d
I20260812 14:54:29.562134 31388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 14:54:29.562454 31388 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3ea7a259ad google::LogMessage::Fail() at ??:0
@ 0x7f3ea7a2888a google::LogMessage::SendToLog() at ??:0
@ 0x7f3ea7a254de google::LogMessage::Flush() at ??:0
@ 0x7f3ea7a26a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3eaf9460f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f3eaf94f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ea98f30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3ea98e8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3eaa74c6db start_thread at ??:0
@ 0x7f3ea652361f clone at ??:0
W20260812 14:54:29.639762 31365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:29.639941 31365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:29.639973 31365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:29.641062 31365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:29.641117 31365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:29.641153 31365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:54:29.642620 31365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:37679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=35417
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:29.642833 31365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:29.643025 31365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:29.647643 31423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.647840 31365 server_base.cc:1034] running on GCE node
W20260812 14:54:29.647744 31421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.647653 31426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.648144 31365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:29.648496 31365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:29.649647 31365 hybrid_clock.cc:648] HybridClock initialized: now 1786546469649632 us; error 43 us; skew 500 ppm
Aug 12 14:54:29 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:29.651945 31365 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:54:29.652359 31365 webserver.cc:460] Webserver started at http://127.4.27.1:35417/ using document root <none> and password file <none>
I20260812 14:54:29.652566 31365 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:29.652608 31365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:29.653885 31365 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:54:29.654546 31435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.654736 31365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:29.654811 31365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "aa575bd50168442e99f224acaba4b49c"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "62a20c86b186ec3a40aefc2c2d3eac98"
server_key_iv: "6d69a47af94a8135cfbfc01509b48fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:29.655126 31365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:29.671631 31365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:29.673593 31365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:29.673722 31365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:29.673920 31365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:29.674324 31446 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:54:29.676270 31365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:54:29.676327 31365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:54:29.676362 31365 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:54:29.676963 31365 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:54:29.676996 31365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:54:29.677084 31446 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Bootstrap starting.
I20260812 14:54:29.683629 31365 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37679
I20260812 14:54:29.683763 31648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37679 every 8 connection(s)
I20260812 14:54:29.684011 31365 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:54:29.687997 31446 log.cc:824] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:29.689065 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 31365
I20260812 14:54:29.689303 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:35821
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:42449
--webserver_port=45621
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:29.690580 31446 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:54:29.690881 31446 tablet_bootstrap.cc:492] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Bootstrap complete.
I20260812 14:54:29.691987 31446 ts_tablet_manager.cc:1397] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260812 14:54:29.693372 31446 raft_consensus.cc:348] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.693686 31446 raft_consensus.cc:729] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa575bd50168442e99f224acaba4b49c, State: Initialized, Role: FOLLOWER
I20260812 14:54:29.693852 31446 consensus_queue.cc:260] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:29.694116 31446 ts_tablet_manager.cc:1428] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260812 14:54:29.775096 31658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:29.775302 31658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:29.775333 31658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:29.776455 31658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:29.776494 31658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:29.776530 31658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
W20260812 14:54:29.777383 31656 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:54:29.684663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:56057)
0812 14:54:29.684826 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.684859 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:29.685261 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:29.777185 (+ 91924us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
W20260812 14:54:29.777616 31652 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:42449 (0 consecutive failures): Network error: Failed to ping master at 127.4.27.62:42449: Client connection negotiation failed: client connection to 127.4.27.62:42449: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:54:29.777918 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.777810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:54819)
0812 14:54:29.777858 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.777896 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:54:29.778193 31656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.778117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:51055)
0812 14:54:29.778153 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.778176 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:54:29.778280 31658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:35821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=45621
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
W20260812 14:54:29.778383 31652 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:54:29.778504 31658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:29.778676 31658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:29.783138 31669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.783119 31674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.783161 31658 server_base.cc:1034] running on GCE node
W20260812 14:54:29.783115 31668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.783464 31658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:29.783675 31658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:29.784806 31658 hybrid_clock.cc:648] HybridClock initialized: now 1786546469784798 us; error 25 us; skew 500 ppm
Aug 12 14:54:29 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:29.787240 31658 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:54:29.787621 31658 webserver.cc:460] Webserver started at http://127.4.27.2:45621/ using document root <none> and password file <none>
I20260812 14:54:29.787830 31658 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:29.787875 31658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:29.789100 31658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:54:29.789654 31687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.789788 31658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:54:29.789860 31658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "b7d795e8db554fff95a01a9c6fb4931a"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "877c79b4bc764389eb33b8f778c123bc"
server_key_iv: "6240466ea6d66fef453a99353bf776bd"
server_key_version: "encryptionkey@0"
I20260812 14:54:29.790166 31658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:29.807497 31658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:29.809160 31658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:29.809291 31658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:29.809486 31658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:29.809753 31658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:29.809787 31658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.809814 31658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:29.809837 31658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.815379 31658 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:35821
I20260812 14:54:29.815445 31901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:35821 every 8 connection(s)
I20260812 14:54:29.815726 31658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:54:29.816645 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.816304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:42385)
0812 14:54:29.816447 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.816512 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260812 14:54:29.816916 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 31658
W20260812 14:54:29.817013 31903 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:42449 (0 consecutive failures): Network error: Failed to ping master at 127.4.27.62:42449: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
I20260812 14:54:29.817209 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:45233
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:42449
--webserver_port=40423
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:54:29.817258 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.817181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:53895)
0812 14:54:29.817219 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.817243 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:54:29.817562 31909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.817511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:35233)
0812 14:54:29.817532 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.817550 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260812 14:54:29.817651 31903 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:54:29.897832 31912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:29.898016 31912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:29.898039 31912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:29.899168 31912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:29.899207 31912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:29.899235 31912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:54:29.900786 31912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:45233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=40423
--tserver_master_addrs=127.4.27.62:42449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:29.900996 31912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:29.901224 31912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:29.905884 31917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.905928 31919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:29.905882 31921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:29.906086 31912 server_base.cc:1034] running on GCE node
I20260812 14:54:29.906244 31912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:29.906492 31912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:29.907626 31912 hybrid_clock.cc:648] HybridClock initialized: now 1786546469907616 us; error 28 us; skew 500 ppm
Aug 12 14:54:29 dist-test-slave-6zbq krb5kdc[29156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:29.909920 31912 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:54:29.910413 31912 webserver.cc:460] Webserver started at http://127.4.27.3:40423/ using document root <none> and password file <none>
I20260812 14:54:29.910679 31912 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:29.910746 31912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:29.912146 31912 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:54:29.912719 31933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.912875 31912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:54:29.912958 31912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "4899ce61dda54e858ecc37628336e945"
format_stamp: "Formatted at 2026-08-12 14:54:28 on dist-test-slave-6zbq"
server_key: "615556fefd4fdf03dc0aa3a77a4ae17b"
server_key_iv: "23c4884dd9946c1fa70fcfcb2dc54103"
server_key_version: "encryptionkey@0"
I20260812 14:54:29.913267 31912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:29.932348 31912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:29.934077 31912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:29.934202 31912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:29.934408 31912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:29.934675 31912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:29.934710 31912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.934743 31912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:29.934765 31912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:29.940074 31912 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:45233
I20260812 14:54:29.940132 32135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:45233 every 8 connection(s)
I20260812 14:54:29.940435 31912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:54:29.941658 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.941317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:39499)
0812 14:54:29.941457 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.941520 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
W20260812 14:54:29.942018 32136 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:42449 (0 consecutive failures): Network error: Failed to ping master at 127.4.27.62:42449: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
I20260812 14:54:29.942245 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.942165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:48761)
0812 14:54:29.942202 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.942226 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260812 14:54:29.942425 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 31912
I20260812 14:54:29.942549 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:29.942464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:35137)
0812 14:54:29.942501 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:29.942526 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:54:29.942652 32136 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:54:30.778996 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:30.778722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:49679)
0812 14:54:30.778885 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:30.778922 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260812 14:54:30.818205 32147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:30.817931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:39499)
0812 14:54:30.818110 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:30.818151 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260812 14:54:30.943090 32148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:30.942918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:33811)
0812 14:54:30.943016 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:30.943047 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260812 14:54:31.088320 32150 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:31.088471 32150 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:31.088783 32150 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233), b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
Aug 12 14:54:31 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.2@KRBTEST.COM
Aug 12 14:54:31 dist-test-slave-6zbq krb5kdc[29156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546469, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.3@KRBTEST.COM
I20260812 14:54:31.098677 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.088992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:43389 (local address 127.4.27.3:45233)
0812 14:54:31.089176 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:31.089189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:31.089221 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:54:31.089678 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:31.089680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:31.089724 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:31.089851 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:31.090780 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.091648 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.092563 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.092963 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.093105 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:31.095498 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:31.095502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:31.095502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:31.095514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:31.097317 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097631 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:31.097634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:31.097634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:31.097721 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097887 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:31.097888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:31.097888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:31.098148 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:31.098212 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:31.098433 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 14:54:31.098563 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260812 14:54:31.098673 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.089100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:33695 (local address 127.4.27.2:35821)
0812 14:54:31.089233 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:31.089241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:31.089270 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:54:31.089686 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:31.089688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:31.089724 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:31.089878 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:31.090780 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.091773 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.092564 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.092966 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.093058 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:31.095135 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:31.095139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:31.095139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:31.095149 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:31.097001 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097413 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:31.097416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:31.097417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:31.097503 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097746 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:31.097747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:31.097747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:31.098042 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:31.098123 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:31.098433 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 14:54:31.098563 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260812 14:54:31.098675 32146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.088894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:45233 (local address 127.4.27.1:43389)
0812 14:54:31.088946 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:31.088956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:31.089011 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:31.089903 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:31.089906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:31.090468 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:31.090662 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:31.090668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.091743 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.091744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:31.092481 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:31.092483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.093043 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.093044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:31.093152 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:31.093724 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:31.093729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:31.095386 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:31.097399 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:31.097400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:31.097554 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:31.097782 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:31.097783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:31.097834 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:31.098205 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:31.098206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:31.098243 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:54:31.098381 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 14:54:31.098547 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":36}
I20260812 14:54:31.098721 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.089011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35821 (local address 127.4.27.1:33695)
0812 14:54:31.089096 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:31.089105 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:31.089143 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:31.089904 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:31.089907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:31.090466 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:31.090661 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:31.090668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.091870 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.091872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:31.092480 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:31.092482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:31.093076 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:31.093077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:31.093153 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:31.093663 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:31.093671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:31.095035 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:31.097089 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:31.097098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:31.097328 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:31.097561 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:31.097562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:31.097562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:31.097685 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:31.098107 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:31.098113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:31.098243 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 14:54:31.098381 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 14:54:31.098610 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":93,"thread_start_us":51,"threads_started":1}
W20260812 14:54:31.099411 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
W20260812 14:54:31.099841 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:31.099884 31441 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa575bd50168442e99f224acaba4b49c; no voters: 4899ce61dda54e858ecc37628336e945, b7d795e8db554fff95a01a9c6fb4931a
I20260812 14:54:31.099994 32150 raft_consensus.cc:2726] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:54:31.779623 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.779441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:35235)
0812 14:54:31.779547 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:31.779579 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260812 14:54:31.818743 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.818591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:58561)
0812 14:54:31.818683 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:31.818713 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
I20260812 14:54:31.943650 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:31.943440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:33619)
0812 14:54:31.943580 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:31.943616 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260812 14:54:32.735973 32164 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:32.736068 32164 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:32.736227 32164 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233), b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
W20260812 14:54:32.736608 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
W20260812 14:54:32.736686 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:32.736711 31441 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa575bd50168442e99f224acaba4b49c; no voters: 4899ce61dda54e858ecc37628336e945, b7d795e8db554fff95a01a9c6fb4931a
I20260812 14:54:32.736793 32164 raft_consensus.cc:2726] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:54:32.780169 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:32.779971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:42579)
0812 14:54:32.780092 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:32.780125 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260812 14:54:32.819348 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:32.819134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:35473)
0812 14:54:32.819275 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:32.819312 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260812 14:54:32.944164 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:32.943983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:52353)
0812 14:54:32.944100 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:32.944131 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260812 14:54:33.780757 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:33.780541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:45719)
0812 14:54:33.780678 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:33.780713 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260812 14:54:33.819958 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:33.819773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:37083)
0812 14:54:33.819883 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:33.819920 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260812 14:54:33.944686 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:33.944518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:57193)
0812 14:54:33.944619 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:33.944651 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260812 14:54:34.781401 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:34.781207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:37265)
0812 14:54:34.781327 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:34.781357 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260812 14:54:34.820555 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:34.820372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:43727)
0812 14:54:34.820490 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:34.820521 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260812 14:54:34.945236 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:34.945050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:57323)
0812 14:54:34.945167 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:34.945200 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260812 14:54:35.270514 32184 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:35.270607 32184 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:35.270767 32184 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233), b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
W20260812 14:54:35.271128 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
W20260812 14:54:35.271183 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:35.271198 31441 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa575bd50168442e99f224acaba4b49c; no voters: 4899ce61dda54e858ecc37628336e945, b7d795e8db554fff95a01a9c6fb4931a
I20260812 14:54:35.271265 32184 raft_consensus.cc:2726] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:54:35.782011 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:35.781816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:38643)
0812 14:54:35.781939 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:35.781968 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260812 14:54:35.821123 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:35.820940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:60105)
0812 14:54:35.821041 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:35.821074 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260812 14:54:35.945801 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:35.945606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:34427)
0812 14:54:35.945734 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:35.945765 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260812 14:54:36.782673 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:36.782471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:41411)
0812 14:54:36.782586 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:36.782628 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260812 14:54:36.821646 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:36.821484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:59521)
0812 14:54:36.821581 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:36.821614 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260812 14:54:36.946519 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:36.946276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:35317)
0812 14:54:36.946438 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:36.946478 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260812 14:54:37.783278 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:37.783042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:48297)
0812 14:54:37.783194 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:37.783231 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260812 14:54:37.822190 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:37.821985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:55027)
0812 14:54:37.822110 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:37.822150 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260812 14:54:37.947090 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:37.946886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:38831)
0812 14:54:37.947021 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:37.947055 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260812 14:54:38.328954 32197 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:38.329048 32197 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:38.329208 32197 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233), b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
W20260812 14:54:38.329584 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
W20260812 14:54:38.329655 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:38.329674 31441 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa575bd50168442e99f224acaba4b49c; no voters: 4899ce61dda54e858ecc37628336e945, b7d795e8db554fff95a01a9c6fb4931a
I20260812 14:54:38.329749 32197 raft_consensus.cc:2726] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:54:38.783886 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:38.783675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:38603)
0812 14:54:38.783808 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:38.783840 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260812 14:54:38.822853 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:38.822647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:36267)
0812 14:54:38.822769 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:38.822804 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260812 14:54:38.947626 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:38.947454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:36127)
0812 14:54:38.947558 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:38.947589 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260812 14:54:39.784547 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:39.784338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:42723)
0812 14:54:39.784472 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:39.784503 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260812 14:54:39.823460 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:39.823253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:53227)
0812 14:54:39.823391 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:39.823423 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260812 14:54:39.948175 32207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:39.947999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:37481)
0812 14:54:39.948105 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:39.948138 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260812 14:54:40.785197 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:40.784984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:49581)
0812 14:54:40.785113 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:40.785150 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260812 14:54:40.824052 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:40.823870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:35235)
0812 14:54:40.823984 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:40.824017 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260812 14:54:40.948849 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:40.948649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:53423)
0812 14:54:40.948774 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:40.948806 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260812 14:54:41.785808 32219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:41.785610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:56865)
0812 14:54:41.785729 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:41.785763 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260812 14:54:41.824626 32220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:41.824448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:53509)
0812 14:54:41.824553 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:41.824585 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260812 14:54:41.949437 32222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:41.949248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:59569)
0812 14:54:41.949362 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:41.949393 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260812 14:54:42.185309 32223 raft_consensus.cc:482] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:42.185411 32223 raft_consensus.cc:504] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4899ce61dda54e858ecc37628336e945" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 45233 } } peers { permanent_uuid: "b7d795e8db554fff95a01a9c6fb4931a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35821 } } peers { permanent_uuid: "aa575bd50168442e99f224acaba4b49c" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37679 } }
I20260812 14:54:42.185546 32223 leader_election.cc:288] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233), b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821)
W20260812 14:54:42.185940 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4899ce61dda54e858ecc37628336e945 (127.4.27.3:45233): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
W20260812 14:54:42.185998 31441 leader_election.cc:341] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7d795e8db554fff95a01a9c6fb4931a (127.4.27.2:35821): Not found: Tablet not found: 6bb5f603c4bb468eb691763dfcd0aa20
I20260812 14:54:42.186014 31441 leader_election.cc:302] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa575bd50168442e99f224acaba4b49c; no voters: 4899ce61dda54e858ecc37628336e945, b7d795e8db554fff95a01a9c6fb4931a
I20260812 14:54:42.186100 32223 raft_consensus.cc:2726] T 6bb5f603c4bb468eb691763dfcd0aa20 P aa575bd50168442e99f224acaba4b49c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:54:42.786490 32224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:42.786248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:44773)
0812 14:54:42.786409 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:42.786445 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260812 14:54:42.825150 32230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:42.824973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:48803)
0812 14:54:42.825080 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:42.825110 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260812 14:54:42.950018 32231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:42.949803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:56617)
0812 14:54:42.949935 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:42.949972 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260812 14:54:43.787083 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:43.786866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:36085)
0812 14:54:43.787000 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:43.787034 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260812 14:54:43.825754 32233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:43.825544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:32933)
0812 14:54:43.825665 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:43.825708 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260812 14:54:43.950596 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:43.950414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:35793)
0812 14:54:43.950525 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:43.950553 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260812 14:54:44.787770 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:44.787549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.1:43219)
0812 14:54:44.787690 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:44.787724 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260812 14:54:44.826450 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:44.826195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.2:50913)
0812 14:54:44.826370 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:44.826410 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260812 14:54:44.946247 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 31365
I20260812 14:54:44.953940 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:44.953655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:42449 (local address 127.4.27.3:47459)
0812 14:54:44.953864 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:44.953899 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:42449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":70,"threads_started":1}
I20260812 14:54:44.961954 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 31658
I20260812 14:54:44.977463 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 31912
2026-08-12T14:54:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18646 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:54:45 dist-test-slave-6zbq krb5kdc[32246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:54:45 dist-test-slave-6zbq krb5kdc[32246](info): set up 2 sockets
Aug 12 14:54:45 dist-test-slave-6zbq krb5kdc[32246](info): commencing operation
krb5kdc: starting...
W20260812 14:54:46.163915 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:54:46.274492 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39227
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:46.354063 32270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:46.354257 32270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:46.354286 32270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:46.355425 32270 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:54:46.355468 32270 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:54:46.355484 32270 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:54:46.355499 32270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:54:46.355515 32270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:46.355532 32270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:46.355551 32270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:54:46.355576 32270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:54:46.357071 32270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:46.357316 32270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:46.357522 32270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:46.362443 32280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.362573 32270 server_base.cc:1034] running on GCE node
W20260812 14:54:46.362450 32285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:46.362655 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.362916 32270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:46.363193 32270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:46.364362 32270 hybrid_clock.cc:648] HybridClock initialized: now 1786546486364341 us; error 35 us; skew 500 ppm
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:46.366945 32270 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:54:46.367512 32270 webserver.cc:460] Webserver started at http://127.4.27.62:37311/ using document root <none> and password file <none>
I20260812 14:54:46.367758 32270 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:46.367805 32270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:46.367930 32270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:46.368944 32270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "77224a7323ff4c79b859b6264a287472"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "10589f573c174d76e4d3520ab5346bec"
server_key_iv: "744e47ebf1eb963c3c6c396bbf781e08"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.369266 32270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "77224a7323ff4c79b859b6264a287472"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "10589f573c174d76e4d3520ab5346bec"
server_key_iv: "744e47ebf1eb963c3c6c396bbf781e08"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.370512 32270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:46.371197 32299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.371364 32270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:54:46.371446 32270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "77224a7323ff4c79b859b6264a287472"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "10589f573c174d76e4d3520ab5346bec"
server_key_iv: "744e47ebf1eb963c3c6c396bbf781e08"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.371500 32270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:46.389858 32270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:46.391856 32270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:46.391959 32270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:46.395563 32270 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:39227
I20260812 14:54:46.395604 32396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:39227 every 8 connection(s)
I20260812 14:54:46.395907 32270 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:54:46.396679 32400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:46.398725 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 32270
I20260812 14:54:46.398804 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:54:46.398904 32400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap starting.
I20260812 14:54:46.399000 4204 external_mini_cluster.cc:1428] Setting key 3a72b57d163d675ccef978209f1e41c6
I20260812 14:54:46.399505 32400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:46.399857 32400 log.cc:824] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:46.400475 32400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: No bootstrap required, opened a new log
I20260812 14:54:46.402225 32400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:54:46.402371 32400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:46.402417 32400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77224a7323ff4c79b859b6264a287472, State: Initialized, Role: FOLLOWER
I20260812 14:54:46.402560 32400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:54:46.402647 32400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:54:46.402675 32400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:54:46.402726 32400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:46.403362 32400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:54:46.403452 32400 leader_election.cc:302] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77224a7323ff4c79b859b6264a287472; no voters:
I20260812 14:54:46.403616 32400 leader_election.cc:288] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:54:46.403673 32410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:46.403836 32410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 LEADER]: Becoming Leader. State: Replica: 77224a7323ff4c79b859b6264a287472, State: Running, Role: LEADER
I20260812 14:54:46.403892 32400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:54:46.403960 32410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:54:46.404337 32410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77224a7323ff4c79b859b6264a287472. Latest consensus state: current_term: 1 leader_uuid: "77224a7323ff4c79b859b6264a287472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } } }
I20260812 14:54:46.404345 32413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77224a7323ff4c79b859b6264a287472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } } }
I20260812 14:54:46.404508 32413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: This master's current role is: LEADER
I20260812 14:54:46.404495 32410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: This master's current role is: LEADER
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:46.405362 32435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:54:46.406054 32435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:54:46.408762 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.399459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.4.27.62:39227)
0812 14:54:46.399617 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.399623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.399648 (+ 25us) server_negotiation.cc:407] Connection header received
0812 14:54:46.400043 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.400050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.400114 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.400241 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:46.400883 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.401744 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.402521 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.403106 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.403246 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.405815 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:46.405819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:46.405819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:46.405828 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:46.407407 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.407777 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.407779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.407779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.407854 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.408035 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.408035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.408035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.408238 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:46.408346 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.408477 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.408573 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260812 14:54:46.410708 32435 catalog_manager.cc:1347] Generated new cluster ID: ccdbc2dda93245bc988b65cda3ce09cf
I20260812 14:54:46.410766 32435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:54:46.424817 32435 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:54:46.425287 32435 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:54:46.433857 32435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Generated new TSK 0
I20260812 14:54:46.434012 32435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:54:46.466336 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:39227
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:46.558172 32453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:46.558379 32453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:46.558421 32453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:46.559756 32453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:46.559799 32453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:46.559824 32453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:54:46.561237 32453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:46.561411 32453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:46.561573 32453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:46.566224 32467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.566407 32453 server_base.cc:1034] running on GCE node
W20260812 14:54:46.566237 32463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:46.566268 32464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.566696 32453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:46.566936 32453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:46.568079 32453 hybrid_clock.cc:648] HybridClock initialized: now 1786546486568061 us; error 38 us; skew 500 ppm
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:46.570464 32453 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:54:46.570843 32453 webserver.cc:460] Webserver started at http://127.4.27.1:42747/ using document root <none> and password file <none>
I20260812 14:54:46.571033 32453 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:46.571071 32453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:46.571161 32453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:46.572139 32453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "2ef7325b1a2e4309a62fa0f985c28389"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "3ec8c75c7b6687f7993fd7a5102520a6"
server_key_iv: "1f552ae6dd41690121dece1a5bd96edb"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.572461 32453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "2ef7325b1a2e4309a62fa0f985c28389"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "3ec8c75c7b6687f7993fd7a5102520a6"
server_key_iv: "1f552ae6dd41690121dece1a5bd96edb"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.573662 32453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:54:46.574327 32478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.574483 32453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.574551 32453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "2ef7325b1a2e4309a62fa0f985c28389"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "3ec8c75c7b6687f7993fd7a5102520a6"
server_key_iv: "1f552ae6dd41690121dece1a5bd96edb"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.574617 32453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:46.604418 32453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:46.606822 32453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:46.606961 32453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:46.607195 32453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:46.607568 32453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:46.607615 32453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.607643 32453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:46.607661 32453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.614224 32453 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43149
I20260812 14:54:46.614266 32688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43149 every 8 connection(s)
I20260812 14:54:46.614595 32453 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:46.622144 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 32453
I20260812 14:54:46.622255 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:54:46.622474 4204 external_mini_cluster.cc:1428] Setting key 14e2ed76514cadddb315fd8f3a0f0a8c
I20260812 14:54:46.623286 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.615271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:35081 (local address 127.4.27.62:39227)
0812 14:54:46.615332 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.615334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.615516 (+ 182us) server_negotiation.cc:407] Connection header received
0812 14:54:46.615976 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.615977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.615986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.616040 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:46.616918 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.617410 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.618450 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.618674 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.618742 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.620818 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:46.620820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:46.620820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:46.620827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:46.621907 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.622299 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.622300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.622300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.622386 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.622609 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.622610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.622611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.622756 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:46.622829 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.623123 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.623200 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 14:54:46.623486 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.615240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:35081)
0812 14:54:46.615397 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.615436 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.615905 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.616094 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.616097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.616657 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:46.616851 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.616855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.617488 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.617489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.618312 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.618314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.618780 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.618782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.618904 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.619381 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:46.619389 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:46.620732 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:46.621973 (+ 1241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:46.621980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:46.621981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:46.622203 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:46.622447 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:46.622448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:46.622448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:46.622549 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:46.622831 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:46.622839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:46.622992 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.623100 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.623270 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260812 14:54:46.625053 32689 heartbeater.cc:347] Connected to a master server at 127.4.27.62:39227
I20260812 14:54:46.625146 32689 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:46.625331 32689 heartbeater.cc:511] Master 127.4.27.62:39227 requested a full tablet report, sending...
I20260812 14:54:46.625878 32330 ts_manager.cc:194] Registered new tserver with Master: 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149)
I20260812 14:54:46.626725 32330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:35081
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:54:46.645673 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:39227
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:46.724960 32697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:46.725140 32697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:46.725171 32697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:46.726255 32697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:46.726301 32697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:46.726366 32697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:54:46.727813 32697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:46.728024 32697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:46.728214 32697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:46.732779 32712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:46.732753 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.732910 32697 server_base.cc:1034] running on GCE node
W20260812 14:54:46.732755 32710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.733167 32697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:46.733388 32697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:46.734527 32697 hybrid_clock.cc:648] HybridClock initialized: now 1786546486734516 us; error 29 us; skew 500 ppm
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:46.736867 32697 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:54:46.737262 32697 webserver.cc:460] Webserver started at http://127.4.27.2:44705/ using document root <none> and password file <none>
I20260812 14:54:46.737463 32697 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:46.737509 32697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:46.737630 32697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:46.738607 32697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "3969344818b547548bf74902c4492aee"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "32153ad2019ef8e8e785617f8b9c4758"
server_key_iv: "f0e0489616522a7d5b2dbfc5b40b3fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.738920 32697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "3969344818b547548bf74902c4492aee"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "32153ad2019ef8e8e785617f8b9c4758"
server_key_iv: "f0e0489616522a7d5b2dbfc5b40b3fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.740070 32697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:46.740690 32725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.740839 32697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:54:46.740909 32697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "3969344818b547548bf74902c4492aee"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "32153ad2019ef8e8e785617f8b9c4758"
server_key_iv: "f0e0489616522a7d5b2dbfc5b40b3fcb"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.740967 32697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:46.752238 32697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:46.754153 32697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:46.754288 32697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:46.754500 32697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:46.754768 32697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:46.754807 32697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.754835 32697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:46.754853 32697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.760154 32697 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:45259
I20260812 14:54:46.760211 465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:45259 every 8 connection(s)
I20260812 14:54:46.760479 32697 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:46.768754 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.761124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:56901 (local address 127.4.27.62:39227)
0812 14:54:46.761181 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.761184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.761352 (+ 168us) server_negotiation.cc:407] Connection header received
0812 14:54:46.761824 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.761826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.761840 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.761890 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:46.762808 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.763373 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.764325 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.764525 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.764580 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.766429 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:46.766431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:46.766431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:46.766439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:46.767510 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.767883 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.767885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.767885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.767949 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.768167 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.768168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.768169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.768278 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:46.768334 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.768583 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.768665 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:54:46.768903 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.761100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:56901)
0812 14:54:46.761247 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.761280 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.761730 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.761954 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.761957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.762527 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:46.762715 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.762720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.763459 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.763460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.764242 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.764243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.764593 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.764594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.764732 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.765092 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:46.765099 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:46.766318 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:46.767587 (+ 1269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:46.767594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:46.767596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:46.767813 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:46.768003 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:46.768004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:46.768004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:46.768105 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:46.768332 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:46.768338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:46.768467 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.768551 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.768707 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260812 14:54:46.769320 467 heartbeater.cc:347] Connected to a master server at 127.4.27.62:39227
I20260812 14:54:46.769405 467 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:46.769560 467 heartbeater.cc:511] Master 127.4.27.62:39227 requested a full tablet report, sending...
I20260812 14:54:46.769927 32330 ts_manager.cc:194] Registered new tserver with Master: 3969344818b547548bf74902c4492aee (127.4.27.2:45259)
I20260812 14:54:46.770028 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 32697
I20260812 14:54:46.770135 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:54:46.770359 4204 external_mini_cluster.cc:1428] Setting key 183f10f82bb4d2c2cdaf4b55a1b66d72
I20260812 14:54:46.770440 32330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:56901
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:54:46.793146 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:39227
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:54:46.870905 475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:54:46.871085 475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:54:46.871115 475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:54:46.872220 475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:54:46.872260 475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:54:46.872285 475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:54:46.873678 475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:54:46.873843 475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:54:46.874027 475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:54:46.878780 486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:54:46.878803 482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.878892 475 server_base.cc:1034] running on GCE node
W20260812 14:54:46.878779 481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:54:46.879135 475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:54:46.879374 475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:54:46.880539 475 hybrid_clock.cc:648] HybridClock initialized: now 1786546486880524 us; error 26 us; skew 500 ppm
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:54:46.882861 475 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:54:46.883226 475 webserver.cc:460] Webserver started at http://127.4.27.3:42761/ using document root <none> and password file <none>
I20260812 14:54:46.883397 475 fs_manager.cc:359] Metadata directory not provided
I20260812 14:54:46.883437 475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:54:46.883528 475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:54:46.884446 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "be09eff69f0146f485fee41b71676747"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "20bf35bf433b027a4ffeba94f3d1e8b7"
server_key_iv: "c919b7c19d4854efc0ff528603c4bde7"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.884727 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "be09eff69f0146f485fee41b71676747"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "20bf35bf433b027a4ffeba94f3d1e8b7"
server_key_iv: "c919b7c19d4854efc0ff528603c4bde7"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.885854 475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:54:46.886468 500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.886616 475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.886714 475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "be09eff69f0146f485fee41b71676747"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "20bf35bf433b027a4ffeba94f3d1e8b7"
server_key_iv: "c919b7c19d4854efc0ff528603c4bde7"
server_key_version: "encryptionkey@0"
I20260812 14:54:46.886785 475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:54:46.906054 475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:54:46.907871 475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:54:46.907989 475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:54:46.908182 475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:54:46.908481 475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:54:46.908514 475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.908535 475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:54:46.908558 475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:54:46.914237 475 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:34829
I20260812 14:54:46.914307 729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:34829 every 8 connection(s)
I20260812 14:54:46.914660 475 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:54:46.917297 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 475
I20260812 14:54:46.917366 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:54:46.917528 4204 external_mini_cluster.cc:1428] Setting key 0a951f956911285065d490bed9fbc29d
Aug 12 14:54:46 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:54:46.923362 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.915777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:38547 (local address 127.4.27.62:39227)
0812 14:54:46.915824 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.915826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.915835 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:54:46.916413 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.916415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.916428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.916478 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:46.917430 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.917959 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.918968 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.919161 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.919217 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.921075 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:46.921076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:46.921077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:46.921093 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:46.922110 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.922507 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.922508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.922508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.922562 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.922777 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.922778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.922779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.922886 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:46.922943 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.923194 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.923273 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:54:46.923514 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.915483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:38547)
0812 14:54:46.915649 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.915685 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.916264 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.916546 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.916551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.917090 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:54:46.917299 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.917304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.918050 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.918051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.918888 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.918890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.919230 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.919231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.919349 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.919724 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:54:46.919732 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:54:46.920991 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:54:46.922187 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:46.922193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:46.922195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:46.922446 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:46.922615 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:54:46.922616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:54:46.922616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:54:46.922721 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:54:46.922943 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:54:46.922950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:54:46.923075 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.923169 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.923316 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260812 14:54:46.923913 730 heartbeater.cc:347] Connected to a master server at 127.4.27.62:39227
I20260812 14:54:46.924001 730 heartbeater.cc:464] Registering TS with master...
I20260812 14:54:46.924162 730 heartbeater.cc:511] Master 127.4.27.62:39227 requested a full tablet report, sending...
I20260812 14:54:46.924501 32330 ts_manager.cc:194] Registered new tserver with Master: be09eff69f0146f485fee41b71676747 (127.4.27.3:34829)
I20260812 14:54:46.924844 32330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:38547
I20260812 14:54:46.928826 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:54:46.935698 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.930073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.4.27.62:39227)
0812 14:54:46.930111 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.930112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.930174 (+ 62us) server_negotiation.cc:407] Connection header received
0812 14:54:46.930244 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.930244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.930252 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.930295 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:54:46.930799 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.931242 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.931920 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.932102 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.932173 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.933334 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:54:46.933336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:54:46.933336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:54:46.933342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:54:46.934602 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.934970 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.934971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.934971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.935035 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:54:46.935286 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:54:46.935287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:54:46.935287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:54:46.935428 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:54:46.935503 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.935579 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.935638 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:54:46.936995 32330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:54:46.938591 32330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:54:46.945335 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.941136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43149 (local address 127.0.0.1:50568)
0812 14:54:46.941572 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.941581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.941629 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.941914 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.941918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.941921 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:46.942026 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.942028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.943319 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.943320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.944426 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.944428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.945050 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.945051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.945126 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.945139 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.945200 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.945261 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":56,"thread_start_us":78,"threads_started":1}
I20260812 14:54:46.945447 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.941313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.4.27.2:45259)
0812 14:54:46.941781 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.941785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.941793 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:46.941826 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.941828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.941841 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.941905 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:46.942090 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.942905 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.944317 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.945072 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.945175 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.945233 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.945329 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.945369 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":76,"threads_started":1}
I20260812 14:54:46.945533 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.941114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45259 (local address 127.0.0.1:43614)
0812 14:54:46.941376 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.941416 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.941485 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.941916 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.941923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.941925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:46.942009 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.942012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.943003 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.943004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.944237 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.944240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.945192 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.945193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.945261 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.945265 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.945315 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.945351 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":16,"thread_start_us":166,"threads_started":1}
I20260812 14:54:46.945691 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.941406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.4.27.1:43149)
0812 14:54:46.941590 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.941594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.941627 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:54:46.941714 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.941716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.941732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.941992 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:46.942090 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.943218 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.944495 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.944976 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.945039 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.945097 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.945561 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.945608 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260812 14:54:46.945812 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.941114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34829 (local address 127.0.0.1:46436)
0812 14:54:46.941387 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.941416 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.941485 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.941914 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.941918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.941920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:46.942008 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.942010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.943335 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.943336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.944301 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.944305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.945172 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.945173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.945239 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.945243 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.945272 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.945311 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":40,"thread_start_us":224,"threads_started":1}
I20260812 14:54:46.945973 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.941494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.4.27.3:34829)
0812 14:54:46.941643 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.941647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.941656 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:54:46.941692 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.941694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.941710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.941812 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:46.942090 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.943200 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.944394 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.945072 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.945382 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.945583 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.945718 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.945839 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260812 14:54:46.946247 337 tablet_service.cc:1467] Processing CreateTablet for tablet fc7c5e873e2e4c8ca43660f72aa83aed (DEFAULT_TABLE table=test-table [id=32a3492f2c2a4b7fa51eb99abd3a15cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:46.946288 32564 tablet_service.cc:1467] Processing CreateTablet for tablet fc7c5e873e2e4c8ca43660f72aa83aed (DEFAULT_TABLE table=test-table [id=32a3492f2c2a4b7fa51eb99abd3a15cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:46.946636 337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7c5e873e2e4c8ca43660f72aa83aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:46.946636 32564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7c5e873e2e4c8ca43660f72aa83aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:46.948983 765 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap starting.
I20260812 14:54:46.948985 764 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Bootstrap starting.
I20260812 14:54:46.949631 765 tablet_bootstrap.cc:654] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:46.949582 597 tablet_service.cc:1467] Processing CreateTablet for tablet fc7c5e873e2e4c8ca43660f72aa83aed (DEFAULT_TABLE table=test-table [id=32a3492f2c2a4b7fa51eb99abd3a15cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:54:46.949884 764 tablet_bootstrap.cc:654] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:46.949884 597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc7c5e873e2e4c8ca43660f72aa83aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:54:46.950148 765 log.cc:824] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:46.950438 764 log.cc:824] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:46.950856 765 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: No bootstrap required, opened a new log
I20260812 14:54:46.950924 765 ts_tablet_manager.cc:1397] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:46.951184 764 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: No bootstrap required, opened a new log
I20260812 14:54:46.951253 764 ts_tablet_manager.cc:1397] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:54:46.952235 765 raft_consensus.cc:348] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.952384 765 raft_consensus.cc:374] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:46.952417 765 raft_consensus.cc:729] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef7325b1a2e4309a62fa0f985c28389, State: Initialized, Role: FOLLOWER
I20260812 14:54:46.952550 765 consensus_queue.cc:260] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.952917 765 ts_tablet_manager.cc:1428] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:54:46.953032 32689 heartbeater.cc:503] Master 127.4.27.62:39227 was elected leader, sending a full tablet report...
I20260812 14:54:46.953090 774 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Bootstrap starting.
I20260812 14:54:46.953336 764 raft_consensus.cc:348] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.953507 764 raft_consensus.cc:374] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:46.953548 764 raft_consensus.cc:729] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3969344818b547548bf74902c4492aee, State: Initialized, Role: FOLLOWER
I20260812 14:54:46.953650 764 consensus_queue.cc:260] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.953744 774 tablet_bootstrap.cc:654] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Neither blocks nor log segments found. Creating new log.
I20260812 14:54:46.953930 467 heartbeater.cc:503] Master 127.4.27.62:39227 was elected leader, sending a full tablet report...
I20260812 14:54:46.953930 764 ts_tablet_manager.cc:1428] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:54:46.954252 774 log.cc:824] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Log is configured to *not* fsync() on all Append() calls
I20260812 14:54:46.954995 774 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: No bootstrap required, opened a new log
I20260812 14:54:46.955061 774 ts_tablet_manager.cc:1397] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:54:46.956283 774 raft_consensus.cc:348] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.956403 774 raft_consensus.cc:374] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:54:46.956432 774 raft_consensus.cc:729] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be09eff69f0146f485fee41b71676747, State: Initialized, Role: FOLLOWER
I20260812 14:54:46.956532 774 consensus_queue.cc:260] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.956782 730 heartbeater.cc:503] Master 127.4.27.62:39227 was elected leader, sending a full tablet report...
I20260812 14:54:46.956768 774 ts_tablet_manager.cc:1428] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:54:46.979796 780 raft_consensus.cc:482] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:54:46.979907 780 raft_consensus.cc:504] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.980244 780 leader_election.cc:288] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3969344818b547548bf74902c4492aee (127.4.27.2:45259), 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149)
I20260812 14:54:46.982913 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.980392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43149 (local address 127.4.27.3:60557)
0812 14:54:46.980446 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.980456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.980494 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.980627 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.980628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.980630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:46.980705 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.980707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.981367 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.981368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.982104 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.982106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.982679 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.982680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.982755 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.982759 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.982810 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.982850 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 14:54:46.982923 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.980465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:60557 (local address 127.4.27.1:43149)
0812 14:54:46.980500 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.980502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.980512 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:54:46.980541 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.980542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.980553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.980622 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:46.980770 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.981298 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.982185 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.982611 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.982664 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.982673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.982820 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.982856 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:54:46.983009 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.980393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45259 (local address 127.4.27.3:36591)
0812 14:54:46.980569 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:46.980580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:54:46.980635 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:54:46.980780 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:54:46.980781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:54:46.980783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:54:46.980852 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.980854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.981446 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.981447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.982181 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:54:46.982184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.982779 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.982780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:54:46.982850 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.982855 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:54:46.982905 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:54:46.982939 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":11,"thread_start_us":112,"threads_started":1}
I20260812 14:54:46.983059 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:46.980450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:36591 (local address 127.4.27.2:45259)
0812 14:54:46.980493 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:46.980494 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:46.980640 (+ 146us) server_negotiation.cc:407] Connection header received
0812 14:54:46.980691 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:46.980692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:46.980704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:46.980764 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:54:46.980905 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.981376 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.982267 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:46.982690 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:46.982758 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:46.982768 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:46.982919 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 14:54:46.983001 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:54:46.983230 32595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c5e873e2e4c8ca43660f72aa83aed" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef7325b1a2e4309a62fa0f985c28389" is_pre_election: true
I20260812 14:54:46.983310 382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c5e873e2e4c8ca43660f72aa83aed" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3969344818b547548bf74902c4492aee" is_pre_election: true
I20260812 14:54:46.983366 32595 raft_consensus.cc:2454] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 0.
I20260812 14:54:46.983455 382 raft_consensus.cc:2454] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 0.
I20260812 14:54:46.983588 511 leader_election.cc:302] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389, be09eff69f0146f485fee41b71676747; no voters:
I20260812 14:54:46.983718 780 raft_consensus.cc:2781] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:54:46.983788 780 raft_consensus.cc:482] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:54:46.983804 780 raft_consensus.cc:3037] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:46.984455 780 raft_consensus.cc:504] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.984570 780 leader_election.cc:288] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 election: Requested vote from peers 3969344818b547548bf74902c4492aee (127.4.27.2:45259), 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149)
I20260812 14:54:46.984750 382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c5e873e2e4c8ca43660f72aa83aed" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3969344818b547548bf74902c4492aee"
I20260812 14:54:46.984738 32595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc7c5e873e2e4c8ca43660f72aa83aed" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef7325b1a2e4309a62fa0f985c28389"
I20260812 14:54:46.984819 32595 raft_consensus.cc:3037] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:46.984818 382 raft_consensus.cc:3037] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:54:46.986037 382 raft_consensus.cc:2454] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 1.
I20260812 14:54:46.986037 32595 raft_consensus.cc:2454] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 1.
I20260812 14:54:46.986274 507 leader_election.cc:302] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389, 3969344818b547548bf74902c4492aee, be09eff69f0146f485fee41b71676747; no voters:
I20260812 14:54:46.986578 780 raft_consensus.cc:2781] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:54:46.986763 780 raft_consensus.cc:686] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 1 LEADER]: Becoming Leader. State: Replica: be09eff69f0146f485fee41b71676747, State: Running, Role: LEADER
I20260812 14:54:46.986851 780 consensus_queue.cc:237] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:54:46.987635 32330 catalog_manager.cc:5495] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 reported cstate change: term changed from 0 to 1, leader changed from <none> to be09eff69f0146f485fee41b71676747 (127.4.27.3). New cstate: current_term: 1 leader_uuid: "be09eff69f0146f485fee41b71676747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } health_report { overall_health: UNKNOWN } } }
I20260812 14:54:47.002321 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:47.000356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.4.27.3:34829)
0812 14:54:47.000408 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:54:47.000409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:54:47.000417 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:54:47.000447 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:54:47.000448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:54:47.000463 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:54:47.000517 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:54:47.000638 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:47.001076 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:47.001777 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:54:47.001948 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:54:47.001987 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:54:47.002085 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:54:47.002168 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:54:47.002220 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:54:47.002236 (+ 16us) server_negotiation.cc:299] Negotiation successful
0812 14:54:47.002250 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:54:47.003890 32595 raft_consensus.cc:1261] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Refusing update from remote peer be09eff69f0146f485fee41b71676747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:47.003896 382 raft_consensus.cc:1261] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Refusing update from remote peer be09eff69f0146f485fee41b71676747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:54:47.004205 784 consensus_queue.cc:1035] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:47.004323 780 consensus_queue.cc:1035] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:54:47.006821 789 mvcc.cc:205] Tried to move back new op lower bound from 7317694410765852672 to 7317694410698833920. Current Snapshot: MvccSnapshot[applied={T|T < 7317694410765852672}]
I20260812 14:54:47.006992 788 mvcc.cc:205] Tried to move back new op lower bound from 7317694410765852672 to 7317694410698833920. Current Snapshot: MvccSnapshot[applied={T|T < 7317694410765852672}]
I20260812 14:54:47.007045 787 mvcc.cc:205] Tried to move back new op lower bound from 7317694410765852672 to 7317694410698833920. Current Snapshot: MvccSnapshot[applied={T|T < 7317694410765852672}]
I20260812 14:54:47.009090 32330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:47.009198 32330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:54:47.009887 32330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Sending DeleteTablet for 3 replicas of tablet fc7c5e873e2e4c8ca43660f72aa83aed
I20260812 14:54:47.010226 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 32453
I20260812 14:54:47.010222 337 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7c5e873e2e4c8ca43660f72aa83aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:47 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260812 14:54:47.010721 597 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7c5e873e2e4c8ca43660f72aa83aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:47 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:46436
I20260812 14:54:47.017565 807 tablet_replica.cc:330] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: stopping tablet replica
I20260812 14:54:47.017673 807 raft_consensus.cc:2229] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:54:47.017812 807 raft_consensus.cc:2258] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:47.017961 806 tablet_replica.cc:330] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: stopping tablet replica
I20260812 14:54:47.018049 806 raft_consensus.cc:2229] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:54:47.018149 806 raft_consensus.cc:2258] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:54:47.019901 807 ts_tablet_manager.cc:1905] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:47.021287 806 ts_tablet_manager.cc:1905] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:54:47.021759 807 ts_tablet_manager.cc:1918] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:47.021827 807 log.cc:1196] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/fc7c5e873e2e4c8ca43660f72aa83aed
I20260812 14:54:47.022091 807 ts_tablet_manager.cc:1939] T fc7c5e873e2e4c8ca43660f72aa83aed P be09eff69f0146f485fee41b71676747: Deleting consensus metadata
I20260812 14:54:47.022557 32302 catalog_manager.cc:4841] TS be09eff69f0146f485fee41b71676747 (127.4.27.3:34829): tablet fc7c5e873e2e4c8ca43660f72aa83aed (table test-table [id=32a3492f2c2a4b7fa51eb99abd3a15cc]) successfully deleted
I20260812 14:54:47.023136 806 ts_tablet_manager.cc:1918] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:54:47.023221 806 log.cc:1196] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/fc7c5e873e2e4c8ca43660f72aa83aed
I20260812 14:54:47.023488 806 ts_tablet_manager.cc:1939] T fc7c5e873e2e4c8ca43660f72aa83aed P 3969344818b547548bf74902c4492aee: Deleting consensus metadata
I20260812 14:54:47.023928 32300 catalog_manager.cc:4841] TS 3969344818b547548bf74902c4492aee (127.4.27.2:45259): tablet fc7c5e873e2e4c8ca43660f72aa83aed (table test-table [id=32a3492f2c2a4b7fa51eb99abd3a15cc]) successfully deleted
W20260812 14:54:47.026985 32305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:43149 (error 108)
I20260812 14:54:47.027321 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 32697
I20260812 14:54:47.027832 754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:54:47.027395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43149 (local address 127.0.0.1:50582)
0812 14:54:47.027761 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 14:54:47.027799 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
W20260812 14:54:47.063309 32305 catalog_manager.cc:4584] TS 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc7c5e873e2e4c8ca43660f72aa83aed: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43149: connect: Connection refused (error 111)
I20260812 14:54:47.068897 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 475
I20260812 14:54:47.079293 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 32270
I20260812 14:55:07.090860 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39227
--webserver_interface=127.4.27.62
--webserver_port=37311
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:07.171371 809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:07.171559 809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:07.171581 809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:07.172696 809 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:55:07.172734 809 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:55:07.172746 809 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:55:07.172758 809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:07.172782 809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:07.172807 809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:07.172829 809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:07.172851 809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:07.174299 809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=37311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:07.174553 809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:07.174762 809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:07.183019 828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.183014 825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.183014 830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:07.183152 809 server_base.cc:1034] running on GCE node
I20260812 14:55:07.183399 809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:07.183667 809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:07.184839 809 hybrid_clock.cc:648] HybridClock initialized: now 1786546507184823 us; error 31 us; skew 500 ppm
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:07.187561 809 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:55:07.188135 809 webserver.cc:460] Webserver started at http://127.4.27.62:37311/ using document root <none> and password file <none>
I20260812 14:55:07.188380 809 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:07.188441 809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:07.189703 809 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:07.190322 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.190505 809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:07.190593 809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "77224a7323ff4c79b859b6264a287472"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "10589f573c174d76e4d3520ab5346bec"
server_key_iv: "744e47ebf1eb963c3c6c396bbf781e08"
server_key_version: "encryptionkey@0"
I20260812 14:55:07.190984 809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:07.207486 809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:07.209159 809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:07.209270 809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:07.213085 809 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:39227
I20260812 14:55:07.213145 921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:39227 every 8 connection(s)
I20260812 14:55:07.213479 809 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:07.214948 922 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:55:07.215600 922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap starting.
I20260812 14:55:07.216600 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 809
I20260812 14:55:07.216856 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:43149
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:39227
--webserver_port=42747
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:55:07.223819 922 log.cc:824] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:07.225500 922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:55:07.225747 922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap complete.
I20260812 14:55:07.227784 922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:07.228000 922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77224a7323ff4c79b859b6264a287472, State: Initialized, Role: FOLLOWER
I20260812 14:55:07.228168 922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:07.228245 922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:07.228299 922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:07.228345 922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 1 FOLLOWER]: Advancing to term 2
I20260812 14:55:07.229341 922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:07.229432 922 leader_election.cc:302] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77224a7323ff4c79b859b6264a287472; no voters:
I20260812 14:55:07.229588 922 leader_election.cc:288] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 14:55:07.229621 926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 14:55:07.229784 926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 LEADER]: Becoming Leader. State: Replica: 77224a7323ff4c79b859b6264a287472, State: Running, Role: LEADER
I20260812 14:55:07.229846 922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:07.229880 926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:07.230157 932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "77224a7323ff4c79b859b6264a287472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } } }
I20260812 14:55:07.230163 926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77224a7323ff4c79b859b6264a287472. Latest consensus state: current_term: 2 leader_uuid: "77224a7323ff4c79b859b6264a287472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } } }
I20260812 14:55:07.230253 932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:07.230258 926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:07.230460 939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:07.231304 939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:07.231390 939 catalog_manager.cc:1259] Loaded cluster ID: ccdbc2dda93245bc988b65cda3ce09cf
I20260812 14:55:07.231423 939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:07.232254 939 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:07.232420 939 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Loaded TSK: 0
I20260812 14:55:07.244077 939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Generated new TSK 1
I20260812 14:55:07.244163 939 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:55:07.297175 924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:07.297374 924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:07.297395 924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:07.298532 924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:07.298568 924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:07.298594 924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:07.299994 924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:43149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=42747
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:07.300196 924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:07.300401 924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:07.305001 968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.305090 970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:07.305161 924 server_base.cc:1034] running on GCE node
W20260812 14:55:07.304981 967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:07.305437 924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:07.305735 924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:07.306913 924 hybrid_clock.cc:648] HybridClock initialized: now 1786546507306891 us; error 33 us; skew 500 ppm
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:07.309834 924 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:55:07.310392 924 webserver.cc:460] Webserver started at http://127.4.27.1:42747/ using document root <none> and password file <none>
I20260812 14:55:07.310644 924 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:07.310739 924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:07.312389 924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:55:07.313138 984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.313308 924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:07.313406 924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "2ef7325b1a2e4309a62fa0f985c28389"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "3ec8c75c7b6687f7993fd7a5102520a6"
server_key_iv: "1f552ae6dd41690121dece1a5bd96edb"
server_key_version: "encryptionkey@0"
I20260812 14:55:07.313803 924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:07.326537 924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:07.328574 924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:07.328744 924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:07.329005 924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:07.329452 1002 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:55:07.330925 924 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:55:07.330983 924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:55:07.331012 924 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:55:07.331871 924 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:55:07.331914 924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:55:07.331976 1002 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap starting.
I20260812 14:55:07.339035 924 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43149
I20260812 14:55:07.339464 924 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:07.339854 1173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43149 every 8 connection(s)
I20260812 14:55:07.341540 1002 log.cc:824] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:07.342015 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 924
I20260812 14:55:07.342322 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:45259
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:39227
--webserver_port=44705
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:55:07.344053 1002 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:55:07.344336 1002 tablet_bootstrap.cc:492] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap complete.
I20260812 14:55:07.345387 1002 ts_tablet_manager.cc:1397] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260812 14:55:07.346692 1002 raft_consensus.cc:348] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:07.347025 1002 raft_consensus.cc:729] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef7325b1a2e4309a62fa0f985c28389, State: Initialized, Role: FOLLOWER
I20260812 14:55:07.347184 1002 consensus_queue.cc:260] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.347448 1002 ts_tablet_manager.cc:1428] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260812 14:55:07.350473 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.340122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:33593)
0812 14:55:07.340409 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.340454 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.340995 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.341687 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.341691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.342016 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:07.342360 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.342367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.344075 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.344076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.344931 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.344934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.345552 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.345554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.345665 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.346109 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:07.346117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:07.347438 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:07.349042 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:07.349049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:07.349051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:07.349255 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:07.349468 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:07.349469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:07.349469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:07.349569 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:07.349886 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:07.349892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:07.350015 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.350097 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.350256 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20260812 14:55:07.350485 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.340218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:33593 (local address 127.4.27.62:39227)
0812 14:55:07.341212 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.341221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.341259 (+ 38us) server_negotiation.cc:407] Connection header received
0812 14:55:07.341382 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.341388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.341441 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.341579 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:07.342656 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.343920 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.345010 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.345409 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.345512 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.347558 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:07.347561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:07.347561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:07.347571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:07.348967 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.349326 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.349329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.349329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.349410 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.349625 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.349625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.349626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.349822 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:07.349893 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.350159 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.350282 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":85,"threads_started":1}
I20260812 14:55:07.351420 1174 heartbeater.cc:347] Connected to a master server at 127.4.27.62:39227
I20260812 14:55:07.351506 1174 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:07.351655 1174 heartbeater.cc:511] Master 127.4.27.62:39227 requested a full tablet report, sending...
I20260812 14:55:07.352280 868 ts_manager.cc:194] Registered new tserver with Master: 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149)
I20260812 14:55:07.353173 868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:33593
I20260812 14:55:07.353932 1174 heartbeater.cc:503] Master 127.4.27.62:39227 was elected leader, sending a full tablet report...
I20260812 14:55:07.357169 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.353424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.4.27.1:43149)
0812 14:55:07.354581 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.354584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.354591 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:55:07.354613 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.354614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.354624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.354686 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:07.354853 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.355460 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.356391 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.356807 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.356866 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.356918 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.357072 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.357107 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":66,"threads_started":1}
I20260812 14:55:07.357178 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.352781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43149 (local address 127.0.0.1:49790)
0812 14:55:07.354425 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.354450 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.354496 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.354706 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.354708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.354710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:07.354793 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.354794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.355529 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.355530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.356315 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.356317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.356880 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.356881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.356982 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.356995 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.357052 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.357105 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"thread_start_us":86,"threads_started":1}
I20260812 14:55:07.357722 1068 tablet_service.cc:1514] Processing DeleteTablet for tablet fc7c5e873e2e4c8ca43660f72aa83aed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:54:47 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:49790
I20260812 14:55:07.358428 1198 tablet_replica.cc:330] stopping tablet replica
I20260812 14:55:07.358517 1198 raft_consensus.cc:2229] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:07.358582 1198 raft_consensus.cc:2258] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:07.358839 1198 ts_tablet_manager.cc:1905] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:07.360725 1198 ts_tablet_manager.cc:1918] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:07.360790 1198 log.cc:1196] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal/wals/fc7c5e873e2e4c8ca43660f72aa83aed
I20260812 14:55:07.360984 1198 ts_tablet_manager.cc:1939] T fc7c5e873e2e4c8ca43660f72aa83aed P 2ef7325b1a2e4309a62fa0f985c28389: Deleting consensus metadata
I20260812 14:55:07.361433 850 catalog_manager.cc:4841] TS 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149): tablet fc7c5e873e2e4c8ca43660f72aa83aed (table test-table [id=32a3492f2c2a4b7fa51eb99abd3a15cc]) successfully deleted
W20260812 14:55:07.435647 1182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:07.435832 1182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:07.435853 1182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:07.436894 1182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:07.436928 1182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:07.436950 1182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:07.438313 1182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:45259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=44705
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:07.438532 1182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:07.438694 1182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:55:07.443195 1182 server_base.cc:1034] running on GCE node
W20260812 14:55:07.443167 1212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.443178 1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.443176 1207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:07.443521 1182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:07.443735 1182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:07.444877 1182 hybrid_clock.cc:648] HybridClock initialized: now 1786546507444867 us; error 28 us; skew 500 ppm
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:07.447083 1182 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:55:07.447460 1182 webserver.cc:460] Webserver started at http://127.4.27.2:44705/ using document root <none> and password file <none>
I20260812 14:55:07.447654 1182 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:07.447713 1182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:07.448889 1182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:07.449371 1232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.449512 1182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.449590 1182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "3969344818b547548bf74902c4492aee"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "32153ad2019ef8e8e785617f8b9c4758"
server_key_iv: "f0e0489616522a7d5b2dbfc5b40b3fcb"
server_key_version: "encryptionkey@0"
I20260812 14:55:07.449863 1182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:07.467787 1182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:07.469321 1182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:07.469445 1182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:07.469635 1182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:07.469892 1182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:07.469924 1182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.469950 1182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:07.469967 1182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.475458 1182 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:45259
I20260812 14:55:07.475525 1455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:45259 every 8 connection(s)
I20260812 14:55:07.475824 1182 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:07.483837 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.476456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:47881 (local address 127.4.27.62:39227)
0812 14:55:07.476499 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.476501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.476726 (+ 225us) server_negotiation.cc:407] Connection header received
0812 14:55:07.477170 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.477172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.477191 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.477246 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:07.478090 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.478622 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.479547 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.479755 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.479811 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.481592 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:07.481593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:07.481594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:07.481601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:07.482675 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.483011 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.483012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.483012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.483061 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.483269 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.483270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.483270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.483370 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:07.483423 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.483658 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.483740 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:55:07.483958 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.476448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:47881)
0812 14:55:07.476602 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.476646 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.477078 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.477316 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.477320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.477822 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:07.477994 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.477998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.478706 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.478708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.479463 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.479465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.479824 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.479825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.479945 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.480312 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:07.480319 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:07.481506 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:07.482744 (+ 1238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:07.482750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:07.482752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:07.482947 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:07.483113 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:07.483113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:07.483114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:07.483212 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:07.483423 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:07.483429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:07.483547 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.483629 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.483770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260812 14:55:07.484383 1458 heartbeater.cc:347] Connected to a master server at 127.4.27.62:39227
I20260812 14:55:07.484463 1458 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:07.484612 1458 heartbeater.cc:511] Master 127.4.27.62:39227 requested a full tablet report, sending...
I20260812 14:55:07.484944 868 ts_manager.cc:194] Registered new tserver with Master: 3969344818b547548bf74902c4492aee (127.4.27.2:45259)
I20260812 14:55:07.484946 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 1182
I20260812 14:55:07.485194 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:34829
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:39227
--webserver_port=42761
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:55:07.485320 868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:47881
W20260812 14:55:07.564759 1463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:07.564924 1463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:07.564944 1463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:07.566015 1463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:07.566054 1463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:07.566082 1463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:07.567502 1463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:34829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=42761
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:07.567705 1463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:07.567901 1463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:07.572328 1473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.572440 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:07.572444 1475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:07.572592 1463 server_base.cc:1034] running on GCE node
I20260812 14:55:07.572760 1463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:07.572971 1463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:07.574100 1463 hybrid_clock.cc:648] HybridClock initialized: now 1786546507574090 us; error 24 us; skew 500 ppm
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:07.576339 1463 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:55:07.577066 1463 webserver.cc:460] Webserver started at http://127.4.27.3:42761/ using document root <none> and password file <none>
I20260812 14:55:07.577280 1463 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:07.577345 1463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:07.578631 1463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:07.579144 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.579308 1463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.579387 1463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "be09eff69f0146f485fee41b71676747"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "20bf35bf433b027a4ffeba94f3d1e8b7"
server_key_iv: "c919b7c19d4854efc0ff528603c4bde7"
server_key_version: "encryptionkey@0"
I20260812 14:55:07.579681 1463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:07.589016 1463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:07.590643 1463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:07.590765 1463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:07.590945 1463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:07.591200 1463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:07.591233 1463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.591259 1463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:07.591281 1463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.596669 1463 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:34829
I20260812 14:55:07.596704 1700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:34829 every 8 connection(s)
I20260812 14:55:07.596988 1463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:55:07.600363 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 1463
Aug 12 14:55:07 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:07.605636 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.597609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:42141 (local address 127.4.27.62:39227)
0812 14:55:07.597671 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.597673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.597835 (+ 162us) server_negotiation.cc:407] Connection header received
0812 14:55:07.598248 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.598249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.598258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.598308 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:07.599136 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.599606 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.600644 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.600918 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.600982 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.602999 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:07.603000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:07.603001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:07.603008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:07.604308 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.604749 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.604751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.604752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.604815 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.605072 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.605073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.605073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.605176 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:07.605235 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.605500 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.605576 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260812 14:55:07.605832 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.597589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:42141)
0812 14:55:07.597728 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.597765 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.598179 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.598388 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.598391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.598896 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:07.599067 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.599071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.599661 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.599661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.600559 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.600561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.601012 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.601013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.601179 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.601602 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:07.601612 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:07.602929 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:07.604380 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:07.604385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:07.604388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:07.604685 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:07.604881 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:07.604881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:07.604882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:07.605008 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:07.605237 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:07.605242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:07.605369 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.605479 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.605627 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260812 14:55:07.606253 1701 heartbeater.cc:347] Connected to a master server at 127.4.27.62:39227
I20260812 14:55:07.606355 1701 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:07.606515 1701 heartbeater.cc:511] Master 127.4.27.62:39227 requested a full tablet report, sending...
I20260812 14:55:07.606863 868 ts_manager.cc:194] Registered new tserver with Master: be09eff69f0146f485fee41b71676747 (127.4.27.3:34829)
I20260812 14:55:07.607125 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.600754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.4.27.62:39227)
0812 14:55:07.600882 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.600884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.600944 (+ 60us) server_negotiation.cc:407] Connection header received
0812 14:55:07.600999 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.600999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.601010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.601066 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:07.601759 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.602213 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.603054 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.603289 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.603353 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.604480 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:07.604480 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:07.604481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:07.604487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:07.605990 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.606421 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.606422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.606422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.606477 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.606658 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.606659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.606660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.606775 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:07.606928 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.606963 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.607039 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260812 14:55:07.607237 868 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:42141
I20260812 14:55:07.607791 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:07.610359 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.608089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.4.27.62:39227)
0812 14:55:07.608121 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.608123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.608236 (+ 113us) server_negotiation.cc:407] Connection header received
0812 14:55:07.608303 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.608304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.608336 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.608392 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:07.608501 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.608928 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.609639 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.609838 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.609889 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.610033 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:07.610075 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 14:55:07.610303 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33132: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:55:07.610409 1706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33132: token expired
W20260812 14:55:07.610433 736 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.27.62:39227)
I20260812 14:55:07.615326 1706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.610656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.4.27.62:39227)
0812 14:55:07.610688 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.610690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.610713 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:55:07.610776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.610777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.610785 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.610827 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:07.611229 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.611642 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.612270 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.612461 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.612511 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.613389 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:07.613390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:07.613390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:07.613396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:07.614398 (+ 1002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.614682 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.614682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.614683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.614732 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:07.614887 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:07.614888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:07.614888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:07.615004 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:07.615081 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.615190 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.615253 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:55:07.616248 868 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:07.617648 868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:07.619477 1068 tablet_service.cc:1467] Processing CreateTablet for tablet 0a977df94ced4b43bc95163ebed7f1ce (DEFAULT_TABLE table=test-table [id=7a9fe3df550a4175a8adb6364465869f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:07.619663 1068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a977df94ced4b43bc95163ebed7f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:07.621122 1002 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap starting.
I20260812 14:55:07.621407 1002 tablet_bootstrap.cc:654] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:07.621865 1002 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: No bootstrap required, opened a new log
I20260812 14:55:07.621907 1002 ts_tablet_manager.cc:1397] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260812 14:55:07.622013 1002 raft_consensus.cc:348] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.622071 1002 raft_consensus.cc:374] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:07.622117 1002 raft_consensus.cc:729] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef7325b1a2e4309a62fa0f985c28389, State: Initialized, Role: FOLLOWER
I20260812 14:55:07.622164 1002 consensus_queue.cc:260] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.622253 1002 ts_tablet_manager.cc:1428] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:07.623014 1196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.619232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34829 (local address 127.0.0.1:53738)
0812 14:55:07.619275 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.619284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.619311 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.619728 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.619730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.619731 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:07.619765 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.619766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.620686 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.620687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.621676 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.621679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.622784 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.622784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.622848 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.622852 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.622908 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.622947 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:55:07.623027 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.619458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.4.27.3:34829)
0812 14:55:07.619599 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.619603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.619611 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:55:07.619639 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.619641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.619652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.619721 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:07.619836 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.620622 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.621766 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.622682 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.622781 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.622837 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.622924 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.622960 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260812 14:55:07.623075 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.619254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45259 (local address 127.0.0.1:58272)
0812 14:55:07.619365 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.619376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.619424 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.619816 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.619817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.619819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:07.619850 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.619851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.620965 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.620966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.622004 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.622007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.622852 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.622853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.622907 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.622909 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.622950 (+ 41us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.622980 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260812 14:55:07.623081 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.619457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.4.27.2:45259)
0812 14:55:07.619671 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.619675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.619685 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:55:07.619723 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.619725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.619740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.619819 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:07.619904 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.620881 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.622087 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.622795 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.622853 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.622901 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.622972 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.623010 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260812 14:55:07.623597 1583 tablet_service.cc:1467] Processing CreateTablet for tablet 0a977df94ced4b43bc95163ebed7f1ce (DEFAULT_TABLE table=test-table [id=7a9fe3df550a4175a8adb6364465869f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:07.623919 1583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a977df94ced4b43bc95163ebed7f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:07.623970 1321 tablet_service.cc:1467] Processing CreateTablet for tablet 0a977df94ced4b43bc95163ebed7f1ce (DEFAULT_TABLE table=test-table [id=7a9fe3df550a4175a8adb6364465869f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:07.624281 1321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a977df94ced4b43bc95163ebed7f1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:07.627324 1727 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Bootstrap starting.
I20260812 14:55:07.627606 1728 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Bootstrap starting.
I20260812 14:55:07.628111 1727 tablet_bootstrap.cc:654] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:07.628175 1728 tablet_bootstrap.cc:654] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:07.628705 1728 log.cc:824] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:07.628696 1727 log.cc:824] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:07.629393 1727 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: No bootstrap required, opened a new log
I20260812 14:55:07.629463 1727 ts_tablet_manager.cc:1397] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:07.629607 1728 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: No bootstrap required, opened a new log
I20260812 14:55:07.629669 1728 ts_tablet_manager.cc:1397] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:07.631347 1727 raft_consensus.cc:348] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.631520 1727 raft_consensus.cc:374] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:07.631567 1727 raft_consensus.cc:729] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be09eff69f0146f485fee41b71676747, State: Initialized, Role: FOLLOWER
I20260812 14:55:07.631590 1728 raft_consensus.cc:348] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.631681 1727 consensus_queue.cc:260] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.631752 1728 raft_consensus.cc:374] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:07.631798 1728 raft_consensus.cc:729] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3969344818b547548bf74902c4492aee, State: Initialized, Role: FOLLOWER
I20260812 14:55:07.631907 1728 consensus_queue.cc:260] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.631943 1727 ts_tablet_manager.cc:1428] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:55:07.631951 1701 heartbeater.cc:503] Master 127.4.27.62:39227 was elected leader, sending a full tablet report...
I20260812 14:55:07.632160 1728 ts_tablet_manager.cc:1428] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:07.632185 1458 heartbeater.cc:503] Master 127.4.27.62:39227 was elected leader, sending a full tablet report...
W20260812 14:55:07.726322 1460 tablet.cc:2369] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:07.787321 1735 raft_consensus.cc:482] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:07.787426 1735 raft_consensus.cc:504] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.787796 1735 leader_election.cc:288] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3969344818b547548bf74902c4492aee (127.4.27.2:45259), 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149)
I20260812 14:55:07.790462 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.788010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:58453 (local address 127.4.27.2:45259)
0812 14:55:07.788071 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.788073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.788080 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:55:07.788108 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.788109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.788123 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.788178 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:07.788333 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.788819 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.789725 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.790110 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.790162 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.790171 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.790384 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.790414 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 14:55:07.790495 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.787940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45259 (local address 127.4.27.3:58453)
0812 14:55:07.787991 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.788002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.788040 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.788195 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.788196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.788198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:07.788270 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.788272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.788885 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.788886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.789637 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.789639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.790200 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.790201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.790282 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.790287 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.790376 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.790421 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:55:07.790836 1352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a977df94ced4b43bc95163ebed7f1ce" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3969344818b547548bf74902c4492aee" is_pre_election: true
I20260812 14:55:07.790987 1352 raft_consensus.cc:2454] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 0.
I20260812 14:55:07.791236 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.787940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43149 (local address 127.4.27.3:40135)
0812 14:55:07.788094 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.788103 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:07.788172 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:07.788290 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:07.788290 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:07.788307 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:07.788371 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.788372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.788977 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.788978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.789674 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:07.789675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.791019 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.791020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:07.791094 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.791099 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:55:07.791144 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:55:07.791179 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":10,"thread_start_us":81,"threads_started":1}
I20260812 14:55:07.791273 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.787985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:40135 (local address 127.4.27.1:43149)
0812 14:55:07.788028 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.788030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.788177 (+ 147us) server_negotiation.cc:407] Connection header received
0812 14:55:07.788214 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.788215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.788227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.788282 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:07.788428 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.788913 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.789763 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.790949 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.791017 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.791028 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.791166 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.791211 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:55:07.791221 1498 leader_election.cc:302] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3969344818b547548bf74902c4492aee, be09eff69f0146f485fee41b71676747; no voters:
I20260812 14:55:07.791359 1735 raft_consensus.cc:2781] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:07.791420 1735 raft_consensus.cc:482] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:07.791448 1735 raft_consensus.cc:3037] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:07.791591 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a977df94ced4b43bc95163ebed7f1ce" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef7325b1a2e4309a62fa0f985c28389" is_pre_election: true
I20260812 14:55:07.791750 1101 raft_consensus.cc:2454] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 0.
I20260812 14:55:07.792234 1735 raft_consensus.cc:504] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.792346 1735 leader_election.cc:288] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 election: Requested vote from peers 3969344818b547548bf74902c4492aee (127.4.27.2:45259), 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149)
I20260812 14:55:07.792474 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a977df94ced4b43bc95163ebed7f1ce" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef7325b1a2e4309a62fa0f985c28389"
I20260812 14:55:07.792500 1352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a977df94ced4b43bc95163ebed7f1ce" candidate_uuid: "be09eff69f0146f485fee41b71676747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3969344818b547548bf74902c4492aee"
I20260812 14:55:07.792541 1101 raft_consensus.cc:3037] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:07.792565 1352 raft_consensus.cc:3037] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:07.793200 1101 raft_consensus.cc:2454] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 1.
I20260812 14:55:07.793342 1352 raft_consensus.cc:2454] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be09eff69f0146f485fee41b71676747 in term 1.
I20260812 14:55:07.793345 1504 leader_election.cc:302] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389, be09eff69f0146f485fee41b71676747; no voters:
I20260812 14:55:07.793426 1735 raft_consensus.cc:2781] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:07.793552 1735 raft_consensus.cc:686] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 1 LEADER]: Becoming Leader. State: Replica: be09eff69f0146f485fee41b71676747, State: Running, Role: LEADER
I20260812 14:55:07.793641 1735 consensus_queue.cc:237] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:07.794401 868 catalog_manager.cc:5495] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 reported cstate change: term changed from 0 to 1, leader changed from <none> to be09eff69f0146f485fee41b71676747 (127.4.27.3). New cstate: current_term: 1 leader_uuid: "be09eff69f0146f485fee41b71676747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } health_report { overall_health: UNKNOWN } } }
I20260812 14:55:07.800447 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.798612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.4.27.3:34829)
0812 14:55:07.798654 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:07.798656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:07.798665 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:55:07.798728 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:07.798729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:07.798743 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:07.798797 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:07.798879 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.799291 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.799903 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:07.800066 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:07.800125 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:07.800219 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:07.800300 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:07.800350 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:07.800362 (+ 12us) server_negotiation.cc:299] Negotiation successful
0812 14:55:07.800375 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:55:07.801981 1352 raft_consensus.cc:1261] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Refusing update from remote peer be09eff69f0146f485fee41b71676747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:07.802000 1101 raft_consensus.cc:1261] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Refusing update from remote peer be09eff69f0146f485fee41b71676747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:07.802250 1735 consensus_queue.cc:1035] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:07.802395 1738 consensus_queue.cc:1035] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:07.804780 1745 mvcc.cc:205] Tried to move back new op lower bound from 7317694495954665472 to 7317694495923343360. Current Snapshot: MvccSnapshot[applied={T|T < 7317694495954665472}]
I20260812 14:55:07.805243 1746 mvcc.cc:205] Tried to move back new op lower bound from 7317694495954665472 to 7317694495923343360. Current Snapshot: MvccSnapshot[applied={T|T < 7317694495954665472}]
I20260812 14:55:07.806524 868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:07.806595 868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:07.807226 868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Sending DeleteTablet for 3 replicas of tablet 0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:07.807495 1068 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a977df94ced4b43bc95163ebed7f1ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:07 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:49790
I20260812 14:55:07.807524 1321 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a977df94ced4b43bc95163ebed7f1ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:07 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:58272
I20260812 14:55:07.807524 1583 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a977df94ced4b43bc95163ebed7f1ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:07 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:53738
I20260812 14:55:07.807579 1198 tablet_replica.cc:330] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: stopping tablet replica
I20260812 14:55:07.807621 1198 raft_consensus.cc:2229] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:07.807657 1198 raft_consensus.cc:2258] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:07.807718 1766 tablet_replica.cc:330] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: stopping tablet replica
I20260812 14:55:07.807803 1766 raft_consensus.cc:2229] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:55:07.807886 1766 raft_consensus.cc:2258] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:07.807996 1198 ts_tablet_manager.cc:1905] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:07.808041 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 924
I20260812 14:55:07.808286 1766 ts_tablet_manager.cc:1905] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:07.809939 1766 ts_tablet_manager.cc:1918] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:07.810009 1766 log.cc:1196] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:07.810324 1766 ts_tablet_manager.cc:1939] T 0a977df94ced4b43bc95163ebed7f1ce P be09eff69f0146f485fee41b71676747: Deleting consensus metadata
I20260812 14:55:07.810765 844 catalog_manager.cc:4841] TS be09eff69f0146f485fee41b71676747 (127.4.27.3:34829): tablet 0a977df94ced4b43bc95163ebed7f1ce (table test-table [id=7a9fe3df550a4175a8adb6364465869f]) successfully deleted
I20260812 14:55:07.815265 1767 tablet_replica.cc:330] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: stopping tablet replica
I20260812 14:55:07.815379 1767 raft_consensus.cc:2229] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:07.815493 1767 raft_consensus.cc:2258] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:07.818687 1767 ts_tablet_manager.cc:1905] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:07.819789 1767 ts_tablet_manager.cc:1918] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:07.819852 1767 log.cc:1196] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:07.820086 1767 ts_tablet_manager.cc:1939] T 0a977df94ced4b43bc95163ebed7f1ce P 3969344818b547548bf74902c4492aee: Deleting consensus metadata
I20260812 14:55:07.820420 843 catalog_manager.cc:4841] TS 3969344818b547548bf74902c4492aee (127.4.27.2:45259): tablet 0a977df94ced4b43bc95163ebed7f1ce (table test-table [id=7a9fe3df550a4175a8adb6364465869f]) successfully deleted
W20260812 14:55:07.821442 850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:43149 (error 108)
I20260812 14:55:07.821745 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 1182
I20260812 14:55:07.821892 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:07.821790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:43149 (local address 127.0.0.1:49804)
0812 14:55:07.821837 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:07.821867 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 14:55:07.827045 850 catalog_manager.cc:4584] TS 2ef7325b1a2e4309a62fa0f985c28389 (127.4.27.1:43149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a977df94ced4b43bc95163ebed7f1ce: Network error: Client connection negotiation failed: client connection to 127.4.27.1:43149: connect: Connection refused (error 111)
I20260812 14:55:07.834189 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 1463
I20260812 14:55:07.841451 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 809
I20260812 14:55:27.852710 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39227
--webserver_interface=127.4.27.62
--webserver_port=37311
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:27.934430 1771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:27.934605 1771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:27.934626 1771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:27.935678 1771 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 14:55:27.935707 1771 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 14:55:27.935719 1771 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 14:55:27.935729 1771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:27.935740 1771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:27.935751 1771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:27.935762 1771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:27.935779 1771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:27.937168 1771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=37311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:27.937358 1771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:27.937517 1771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:27.942201 1779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:27.942241 1780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:27.942273 1771 server_base.cc:1034] running on GCE node
W20260812 14:55:27.942238 1782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:27.942615 1771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:27.942870 1771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:27.944005 1771 hybrid_clock.cc:648] HybridClock initialized: now 1786546527943993 us; error 27 us; skew 500 ppm
Aug 12 14:55:27 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546527, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:27.946213 1771 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:55:27.946609 1771 webserver.cc:460] Webserver started at http://127.4.27.62:37311/ using document root <none> and password file <none>
I20260812 14:55:27.946811 1771 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:27.946859 1771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:27.948094 1771 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:27.948701 1798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:27.948848 1771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:27.948928 1771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "77224a7323ff4c79b859b6264a287472"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "10589f573c174d76e4d3520ab5346bec"
server_key_iv: "744e47ebf1eb963c3c6c396bbf781e08"
server_key_version: "encryptionkey@0"
I20260812 14:55:27.949218 1771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:27.959645 1771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:27.961127 1771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:27.961244 1771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:27.964854 1771 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:39227
I20260812 14:55:27.964893 1890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:39227 every 8 connection(s)
I20260812 14:55:27.965193 1771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:27.966529 1891 sys_catalog.cc:263] Verifying existing consensus state
I20260812 14:55:27.967131 1891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap starting.
I20260812 14:55:27.968528 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 1771
I20260812 14:55:27.968871 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:43149
--local_ip_for_outbound_sockets=127.4.27.1
--tserver_master_addrs=127.4.27.62:39227
--webserver_port=42747
--webserver_interface=127.4.27.1
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:55:27.975852 1891 log.cc:824] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:27.977828 1891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:55:27.978077 1891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Bootstrap complete.
I20260812 14:55:27.980086 1891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:27.980311 1891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77224a7323ff4c79b859b6264a287472, State: Initialized, Role: FOLLOWER
I20260812 14:55:27.980463 1891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:27.980533 1891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:27.980574 1891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:27.980623 1891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 2 FOLLOWER]: Advancing to term 3
I20260812 14:55:27.981549 1891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:27.981637 1891 leader_election.cc:302] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77224a7323ff4c79b859b6264a287472; no voters:
I20260812 14:55:27.981786 1891 leader_election.cc:288] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 14:55:27.981842 1898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 14:55:27.981958 1898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [term 3 LEADER]: Becoming Leader. State: Replica: 77224a7323ff4c79b859b6264a287472, State: Running, Role: LEADER
I20260812 14:55:27.982041 1898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } }
I20260812 14:55:27.982040 1891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:27.982707 1902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "77224a7323ff4c79b859b6264a287472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } } }
I20260812 14:55:27.982713 1903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77224a7323ff4c79b859b6264a287472. Latest consensus state: current_term: 3 leader_uuid: "77224a7323ff4c79b859b6264a287472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77224a7323ff4c79b859b6264a287472" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39227 } } }
I20260812 14:55:27.982836 1903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:27.983026 1902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:27.983022 1912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:27.983839 1912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:27.983947 1912 catalog_manager.cc:1259] Loaded cluster ID: ccdbc2dda93245bc988b65cda3ce09cf
I20260812 14:55:27.983984 1912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:27.984814 1912 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:27.984972 1912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Loaded TSK: 0
I20260812 14:55:27.985008 1912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 77224a7323ff4c79b859b6264a287472: Loaded TSK: 1
F20260812 14:55:27.985198 1912 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f57cc2899ad google::LogMessage::Fail() at ??:0
@ 0x7f57cc28c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f57cc2894de google::LogMessage::Flush() at ??:0
@ 0x7f57cc28aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f57ce9f6104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f57ce9f3d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f57d41e31cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f57d41e348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f57d41e3594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f57d41a9c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f57d41b3ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f57ce1570d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f57ce14c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f57cefb06db start_thread at ??:0
@ 0x7f57cad8761f clone at ??:0
W20260812 14:55:28.063640 1894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:28.063824 1894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:28.063856 1894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:28.064951 1894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:28.064992 1894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:28.065026 1894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:28.066490 1894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:43149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=42747
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:28.066706 1894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:28.066901 1894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:28.071663 1974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:28.071681 1977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:28.071671 1971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:28.071884 1894 server_base.cc:1034] running on GCE node
I20260812 14:55:28.072094 1894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:28.072315 1894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:28.073453 1894 hybrid_clock.cc:648] HybridClock initialized: now 1786546528073431 us; error 40 us; skew 500 ppm
Aug 12 14:55:28 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:28.075949 1894 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:55:28.076350 1894 webserver.cc:460] Webserver started at http://127.4.27.1:42747/ using document root <none> and password file <none>
I20260812 14:55:28.076561 1894 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:28.076617 1894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:28.077885 1894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:28.078624 1993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.078778 1894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:28.078861 1894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "2ef7325b1a2e4309a62fa0f985c28389"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "3ec8c75c7b6687f7993fd7a5102520a6"
server_key_iv: "1f552ae6dd41690121dece1a5bd96edb"
server_key_version: "encryptionkey@0"
I20260812 14:55:28.079141 1894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:28.092177 1894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:28.094087 1894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:28.094211 1894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:28.094451 1894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:28.094877 2005 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 14:55:28.096730 1894 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 14:55:28.096796 1894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260812 14:55:28.096829 1894 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 14:55:28.097401 1894 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 14:55:28.097434 1894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260812 14:55:28.097522 2005 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap starting.
I20260812 14:55:28.099361 2005 log.cc:824] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:28.102116 2005 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 14:55:28.102417 2005 tablet_bootstrap.cc:492] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Bootstrap complete.
I20260812 14:55:28.102577 2005 ts_tablet_manager.cc:1397] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260812 14:55:28.104012 1894 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:43149
I20260812 14:55:28.104094 2180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:43149 every 8 connection(s)
I20260812 14:55:28.104357 1894 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:28.104512 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 1894
I20260812 14:55:28.104317 2005 raft_consensus.cc:348] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:28.104791 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:45259
--local_ip_for_outbound_sockets=127.4.27.2
--tserver_master_addrs=127.4.27.62:39227
--webserver_port=44705
--webserver_interface=127.4.27.2
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:55:28.104835 2005 raft_consensus.cc:729] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef7325b1a2e4309a62fa0f985c28389, State: Initialized, Role: FOLLOWER
I20260812 14:55:28.104992 2005 consensus_queue.cc:260] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:28.105244 2005 ts_tablet_manager.cc:1428] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260812 14:55:28.175187 2190 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:55:28.106193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:45227)
0812 14:55:28.106382 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.106427 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:28.106579 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:28.174995 (+ 68416us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
W20260812 14:55:28.175455 2182 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:39227 (0 consecutive failures): Network error: Failed to ping master at 127.4.27.62:39227: Client connection negotiation failed: client connection to 127.4.27.62:39227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 14:55:28.175815 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.175685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:32877)
0812 14:55:28.175740 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.175790 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 14:55:28.176138 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.176055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:58853)
0812 14:55:28.176093 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.176118 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:55:28.176268 2182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 14:55:28.186442 2189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:28.186627 2189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:28.186658 2189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:28.187789 2189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:28.187829 2189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:28.187865 2189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:28.189260 2189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:45259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=44705
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:28.189482 2189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:28.189669 2189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:28.194190 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:28.194190 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:28.194202 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:28.194271 2189 server_base.cc:1034] running on GCE node
I20260812 14:55:28.194664 2189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:28.194876 2189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:28.196000 2189 hybrid_clock.cc:648] HybridClock initialized: now 1786546528195991 us; error 26 us; skew 500 ppm
Aug 12 14:55:28 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:28.198316 2189 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:55:28.198731 2189 webserver.cc:460] Webserver started at http://127.4.27.2:44705/ using document root <none> and password file <none>
I20260812 14:55:28.198918 2189 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:28.198964 2189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:28.200158 2189 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:28.200642 2221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.200799 2189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:28.200879 2189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "3969344818b547548bf74902c4492aee"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "32153ad2019ef8e8e785617f8b9c4758"
server_key_iv: "f0e0489616522a7d5b2dbfc5b40b3fcb"
server_key_version: "encryptionkey@0"
I20260812 14:55:28.201187 2189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:28.232522 2189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:28.234201 2189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:28.234333 2189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:28.236698 2189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:28.236975 2189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:28.237008 2189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.237035 2189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:28.237052 2189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.242704 2189 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:45259
I20260812 14:55:28.242765 2417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:45259 every 8 connection(s)
I20260812 14:55:28.243059 2189 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:55:28.243958 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.243620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:53171)
0812 14:55:28.243767 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.243829 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
W20260812 14:55:28.244334 2418 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:39227 (0 consecutive failures): Network error: Failed to ping master at 127.4.27.62:39227: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
I20260812 14:55:28.244582 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.244503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:41851)
0812 14:55:28.244541 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.244565 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:55:28.244871 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.244795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:58579)
0812 14:55:28.244832 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.244856 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:55:28.244963 2418 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:55:28.252244 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 2189
I20260812 14:55:28.252475 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:34829
--local_ip_for_outbound_sockets=127.4.27.3
--tserver_master_addrs=127.4.27.62:39227
--webserver_port=42761
--webserver_interface=127.4.27.3
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:28.332345 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:28.332517 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:28.332547 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:28.333614 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:28.333652 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:28.333686 2425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:28.335140 2425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:34829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=42761
--tserver_master_addrs=127.4.27.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:28.335361 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:28.335562 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:28.340139 2432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:28.340170 2433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:28.340163 2437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:28.340322 2425 server_base.cc:1034] running on GCE node
I20260812 14:55:28.340487 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:28.340699 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:28.341833 2425 hybrid_clock.cc:648] HybridClock initialized: now 1786546528341820 us; error 26 us; skew 500 ppm
Aug 12 14:55:28 dist-test-slave-6zbq krb5kdc[32246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:28.344022 2425 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:55:28.344396 2425 webserver.cc:460] Webserver started at http://127.4.27.3:42761/ using document root <none> and password file <none>
I20260812 14:55:28.344595 2425 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:28.344641 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:28.345795 2425 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:28.346297 2453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.346460 2425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.346541 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "be09eff69f0146f485fee41b71676747"
format_stamp: "Formatted at 2026-08-12 14:54:46 on dist-test-slave-6zbq"
server_key: "20bf35bf433b027a4ffeba94f3d1e8b7"
server_key_iv: "c919b7c19d4854efc0ff528603c4bde7"
server_key_version: "encryptionkey@0"
I20260812 14:55:28.346818 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:28.363210 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:28.364787 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:28.364908 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:28.365091 2425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:28.365346 2425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:28.365378 2425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.365404 2425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:28.365422 2425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:28.370731 2425 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:34829
I20260812 14:55:28.370772 2662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:34829 every 8 connection(s)
I20260812 14:55:28.371059 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:55:28.371943 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.371604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:41141)
0812 14:55:28.371742 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.371817 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
W20260812 14:55:28.372274 2663 heartbeater.cc:644] Failed to heartbeat to 127.4.27.62:39227 (0 consecutive failures): Network error: Failed to ping master at 127.4.27.62:39227: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
I20260812 14:55:28.372491 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.372416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:59193)
0812 14:55:28.372452 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.372476 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:55:28.372687 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:28.372631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:59413)
0812 14:55:28.372659 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:28.372676 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 14:55:28.372779 2663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 14:55:28.377453 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 2425
I20260812 14:55:29.176759 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.176536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:43873)
0812 14:55:29.176668 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:29.176702 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260812 14:55:29.245457 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.245244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:47351)
0812 14:55:29.245376 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:29.245410 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260812 14:55:29.373240 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.373055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:49197)
0812 14:55:29.373166 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:29.373198 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260812 14:55:29.862068 2674 raft_consensus.cc:482] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:29.862190 2674 raft_consensus.cc:504] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:29.862586 2674 leader_election.cc:288] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be09eff69f0146f485fee41b71676747 (127.4.27.3:34829), 3969344818b547548bf74902c4492aee (127.4.27.2:45259)
Aug 12 14:55:29 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.2@KRBTEST.COM
Aug 12 14:55:29 dist-test-slave-6zbq krb5kdc[32246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.3@KRBTEST.COM
I20260812 14:55:29.874444 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.862763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45259 (local address 127.4.27.1:37181)
0812 14:55:29.862929 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:29.862941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:29.863001 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:29.863791 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:29.863792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:29.864094 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:29.864286 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:29.864292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.865339 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.865340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:29.866097 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:29.866099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.866676 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.866677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:29.866793 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:29.869739 (+ 2946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:29.869749 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:29.871133 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:29.873087 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:29.873097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:29.873296 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:29.873511 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:29.873513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:29.873607 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:29.873940 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:29.873946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:29.874061 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 14:55:29.874145 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:55:29.874309 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":28,"thread_start_us":41,"threads_started":1}
I20260812 14:55:29.874444 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.862888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:37181 (local address 127.4.27.2:45259)
0812 14:55:29.863069 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:29.863078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:29.863097 (+ 19us) server_negotiation.cc:407] Connection header received
0812 14:55:29.863596 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:29.863599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:29.863649 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:29.863784 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:29.864394 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.865239 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.866173 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.866592 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.866690 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:29.871228 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:29.871232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:29.871232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:29.871241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:29.873014 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873368 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:29.873371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:29.873372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:29.873453 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873663 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:29.873663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:29.873664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:29.873874 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:29.873944 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:29.874203 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 14:55:29.874321 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":58,"threads_started":1}
I20260812 14:55:29.874454 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.862901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:49209 (local address 127.4.27.3:34829)
0812 14:55:29.863063 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:29.863075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:29.863105 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:55:29.863525 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:29.863527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:29.863569 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:29.863697 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:29.864543 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.865361 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.866172 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.866592 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.866690 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:29.871624 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:29.871628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:29.871628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:29.871639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:29.873186 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873526 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:29.873529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:29.873529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:29.873611 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873790 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:29.873791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:29.873791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:29.873991 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:29.874056 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:29.874222 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:55:29.874361 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":53,"threads_started":1}
I20260812 14:55:29.874516 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:29.862763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34829 (local address 127.4.27.1:49209)
0812 14:55:29.862880 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:29.862890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:29.862948 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:29.863752 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:29.863755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:29.864070 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:29.864440 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:29.864442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.865442 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.865443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:29.866096 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:29.866098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:29.866672 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:29.866673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:29.866793 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:29.869736 (+ 2943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:29.869745 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:29.871510 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:29.873265 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:29.873266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:29.873452 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:29.873670 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:29.873671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:29.873671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:29.873738 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:29.874054 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:29.874055 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:29.874098 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:55:29.874161 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:55:29.874410 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":103,"thread_start_us":58,"threads_started":1}
W20260812 14:55:29.875770 1995 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3969344818b547548bf74902c4492aee (127.4.27.2:45259): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
W20260812 14:55:29.875871 1996 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be09eff69f0146f485fee41b71676747 (127.4.27.3:34829): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:29.875923 1996 leader_election.cc:302] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389; no voters: 3969344818b547548bf74902c4492aee, be09eff69f0146f485fee41b71676747
I20260812 14:55:29.876031 2674 raft_consensus.cc:2726] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:55:30.177239 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:30.177114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:41573)
0812 14:55:30.177174 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:30.177200 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 14:55:30.246061 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:30.245857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:37595)
0812 14:55:30.245979 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:30.246019 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260812 14:55:30.373821 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:30.373628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:51181)
0812 14:55:30.373750 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:30.373781 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260812 14:55:31.177809 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:31.177593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:55065)
0812 14:55:31.177731 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:31.177764 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260812 14:55:31.246680 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:31.246482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:47227)
0812 14:55:31.246606 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:31.246641 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260812 14:55:31.374368 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:31.374173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:47743)
0812 14:55:31.374280 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:31.374313 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260812 14:55:32.018042 2690 raft_consensus.cc:482] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:32.018141 2690 raft_consensus.cc:504] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:32.018311 2690 leader_election.cc:288] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be09eff69f0146f485fee41b71676747 (127.4.27.3:34829), 3969344818b547548bf74902c4492aee (127.4.27.2:45259)
W20260812 14:55:32.018718 1995 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3969344818b547548bf74902c4492aee (127.4.27.2:45259): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
W20260812 14:55:32.018795 1996 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be09eff69f0146f485fee41b71676747 (127.4.27.3:34829): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:32.018836 1996 leader_election.cc:302] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389; no voters: 3969344818b547548bf74902c4492aee, be09eff69f0146f485fee41b71676747
I20260812 14:55:32.018895 2690 raft_consensus.cc:2726] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:55:32.178419 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:32.178185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:42741)
0812 14:55:32.178306 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:32.178369 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260812 14:55:32.247262 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:32.247067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:44683)
0812 14:55:32.247171 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:32.247209 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260812 14:55:32.374927 2696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:32.374748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:37719)
0812 14:55:32.374866 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:32.374895 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260812 14:55:33.179013 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:33.178812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:49975)
0812 14:55:33.178932 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:33.178967 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260812 14:55:33.247815 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:33.247616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:36141)
0812 14:55:33.247744 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:33.247779 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260812 14:55:33.375479 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:33.375286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:45231)
0812 14:55:33.375409 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:33.375444 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260812 14:55:34.179572 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:34.179375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:44869)
0812 14:55:34.179490 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:34.179522 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260812 14:55:34.248916 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:34.248190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:35247)
0812 14:55:34.248805 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:34.248867 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":537,"threads_started":1}
I20260812 14:55:34.376026 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:34.375842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:37553)
0812 14:55:34.375956 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:34.375988 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260812 14:55:35.180181 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:35.179979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:45697)
0812 14:55:35.180091 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:35.180134 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260812 14:55:35.187640 2712 raft_consensus.cc:482] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:35.187718 2712 raft_consensus.cc:504] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:35.187846 2712 leader_election.cc:288] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be09eff69f0146f485fee41b71676747 (127.4.27.3:34829), 3969344818b547548bf74902c4492aee (127.4.27.2:45259)
W20260812 14:55:35.188185 1995 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3969344818b547548bf74902c4492aee (127.4.27.2:45259): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
W20260812 14:55:35.188256 1996 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be09eff69f0146f485fee41b71676747 (127.4.27.3:34829): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:35.188293 1996 leader_election.cc:302] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389; no voters: 3969344818b547548bf74902c4492aee, be09eff69f0146f485fee41b71676747
I20260812 14:55:35.188359 2712 raft_consensus.cc:2726] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:55:35.249471 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:35.249284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:51631)
0812 14:55:35.249387 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:35.249417 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260812 14:55:35.376605 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:35.376414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:42463)
0812 14:55:35.376534 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:35.376567 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260812 14:55:36.180814 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:36.180590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:42419)
0812 14:55:36.180732 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:36.180768 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260812 14:55:36.250074 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:36.249864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:53925)
0812 14:55:36.249995 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:36.250033 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260812 14:55:36.377122 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:36.376941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:47975)
0812 14:55:36.377050 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:36.377083 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260812 14:55:37.181422 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:37.181209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:36625)
0812 14:55:37.181344 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:37.181378 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260812 14:55:37.250716 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:37.250512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:46877)
0812 14:55:37.250633 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:37.250670 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260812 14:55:37.377671 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:37.377481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:35693)
0812 14:55:37.377598 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:37.377635 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260812 14:55:38.182027 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:38.181813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:36625)
0812 14:55:38.181948 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:38.181983 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260812 14:55:38.251284 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:38.251076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:60661)
0812 14:55:38.251205 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:38.251241 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260812 14:55:38.378216 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:38.378030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:33219)
0812 14:55:38.378148 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:38.378180 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260812 14:55:38.758244 2734 raft_consensus.cc:482] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:38.758364 2734 raft_consensus.cc:504] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be09eff69f0146f485fee41b71676747" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34829 } } peers { permanent_uuid: "3969344818b547548bf74902c4492aee" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45259 } } peers { permanent_uuid: "2ef7325b1a2e4309a62fa0f985c28389" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 43149 } }
I20260812 14:55:38.758519 2734 leader_election.cc:288] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be09eff69f0146f485fee41b71676747 (127.4.27.3:34829), 3969344818b547548bf74902c4492aee (127.4.27.2:45259)
W20260812 14:55:38.758876 1996 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be09eff69f0146f485fee41b71676747 (127.4.27.3:34829): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
W20260812 14:55:38.758970 1995 leader_election.cc:341] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3969344818b547548bf74902c4492aee (127.4.27.2:45259): Not found: Tablet not found: 0a977df94ced4b43bc95163ebed7f1ce
I20260812 14:55:38.759087 1995 leader_election.cc:302] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef7325b1a2e4309a62fa0f985c28389; no voters: 3969344818b547548bf74902c4492aee, be09eff69f0146f485fee41b71676747
I20260812 14:55:38.759169 2734 raft_consensus.cc:2726] T 0a977df94ced4b43bc95163ebed7f1ce P 2ef7325b1a2e4309a62fa0f985c28389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 14:55:39.182619 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:39.182428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:52125)
0812 14:55:39.182540 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:39.182573 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260812 14:55:39.251840 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:39.251663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:57669)
0812 14:55:39.251769 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:39.251801 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260812 14:55:39.378767 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:39.378578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:50261)
0812 14:55:39.378688 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:39.378724 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260812 14:55:40.183202 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:40.183004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:43513)
0812 14:55:40.183118 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:40.183157 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260812 14:55:40.252447 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:40.252192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:42683)
0812 14:55:40.252375 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:40.252409 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":59,"threads_started":1}
I20260812 14:55:40.379321 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:40.379120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:51353)
0812 14:55:40.379251 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:40.379285 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260812 14:55:41.183808 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:41.183618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:45119)
0812 14:55:41.183727 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:41.183756 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260812 14:55:41.253014 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:41.252827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:49231)
0812 14:55:41.252939 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:41.252973 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260812 14:55:41.379850 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:41.379669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:45415)
0812 14:55:41.379781 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:41.379815 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260812 14:55:42.184444 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:42.184238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:59471)
0812 14:55:42.184369 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:42.184399 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260812 14:55:42.253654 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:42.253436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:33447)
0812 14:55:42.253569 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:42.253607 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260812 14:55:42.380525 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:42.380302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:33167)
0812 14:55:42.380441 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:42.380480 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260812 14:55:43.185046 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:43.184848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.1:37255)
0812 14:55:43.184970 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:43.185003 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260812 14:55:43.254331 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:43.254147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.2:45293)
0812 14:55:43.254260 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:43.254293 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260812 14:55:43.378504 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 1894
I20260812 14:55:43.384930 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:43.383357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39227 (local address 127.4.27.3:43347)
0812 14:55:43.384842 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:43.384883 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.62:39227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1467,"thread_start_us":52,"threads_started":1}
I20260812 14:55:43.391520 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 2189
I20260812 14:55:43.404425 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 2425
2026-08-12T14:55:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58424 ms)
[----------] 30 tests from SecurityITest (229606 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:43 dist-test-slave-6zbq krb5kdc[2766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:43 dist-test-slave-6zbq krb5kdc[2766](info): set up 2 sockets
Aug 12 14:55:43 dist-test-slave-6zbq krb5kdc[2766](info): commencing operation
krb5kdc: starting...
W20260812 14:55:44.595741 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:44 dist-test-slave-6zbq krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:55:44.704807 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:44155
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:44155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:44.782361 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:44.782534 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:44.782554 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:44.783589 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:44.783622 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:44.783643 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:44.783653 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:44.783669 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:44.785027 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:44155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:44155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:44.785197 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:44.785351 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:44.789614 2801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:44.789625 2802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:44.789697 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:44.789697 2792 server_base.cc:1034] running on GCE node
I20260812 14:55:44.790010 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:44.790254 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:44.791385 2792 hybrid_clock.cc:648] HybridClock initialized: now 1786546544791368 us; error 34 us; skew 500 ppm
Aug 12 14:55:44 dist-test-slave-6zbq krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546544, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:44.793576 2792 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:55:44.793968 2792 webserver.cc:460] Webserver started at http://127.4.27.62:44915/ using document root <none> and password file <none>
I20260812 14:55:44.794158 2792 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:44.794196 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:44.794279 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:44.795215 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "c540fab613cc415cab05f8f312489ff6"
format_stamp: "Formatted at 2026-08-12 14:55:44 on dist-test-slave-6zbq"
server_key: "e97a8f1a4bb6a88922c6df247dbae4ce"
server_key_iv: "d7192a86a141c552326f86093fa6ccb8"
server_key_version: "encryptionkey@0"
I20260812 14:55:44.795490 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "c540fab613cc415cab05f8f312489ff6"
format_stamp: "Formatted at 2026-08-12 14:55:44 on dist-test-slave-6zbq"
server_key: "e97a8f1a4bb6a88922c6df247dbae4ce"
server_key_iv: "d7192a86a141c552326f86093fa6ccb8"
server_key_version: "encryptionkey@0"
I20260812 14:55:44.796597 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:55:44.797205 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:44.797331 2792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:44.797397 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "c540fab613cc415cab05f8f312489ff6"
format_stamp: "Formatted at 2026-08-12 14:55:44 on dist-test-slave-6zbq"
server_key: "e97a8f1a4bb6a88922c6df247dbae4ce"
server_key_iv: "d7192a86a141c552326f86093fa6ccb8"
server_key_version: "encryptionkey@0"
I20260812 14:55:44.797459 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:44.824302 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:44.825811 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:44.825927 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:44.829612 2792 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:44155
I20260812 14:55:44.829648 2939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:44155 every 8 connection(s)
I20260812 14:55:44.829932 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:44.830708 2941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:44.832787 2941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6: Bootstrap starting.
I20260812 14:55:44.833304 2941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:44.833559 2941 log.cc:824] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:44.834154 2941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6: No bootstrap required, opened a new log
I20260812 14:55:44.835687 2941 raft_consensus.cc:348] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540fab613cc415cab05f8f312489ff6" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44155 } }
I20260812 14:55:44.835817 2941 raft_consensus.cc:374] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:44.835839 2941 raft_consensus.cc:729] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c540fab613cc415cab05f8f312489ff6, State: Initialized, Role: FOLLOWER
I20260812 14:55:44.835973 2941 consensus_queue.cc:260] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540fab613cc415cab05f8f312489ff6" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44155 } }
I20260812 14:55:44.836038 2941 raft_consensus.cc:388] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:44.836062 2941 raft_consensus.cc:482] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:44.836107 2941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:44.836572 2941 raft_consensus.cc:504] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540fab613cc415cab05f8f312489ff6" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44155 } }
I20260812 14:55:44.836668 2941 leader_election.cc:302] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c540fab613cc415cab05f8f312489ff6; no voters:
I20260812 14:55:44.836814 2941 leader_election.cc:288] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:44.836848 2945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:44.836977 2945 raft_consensus.cc:686] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [term 1 LEADER]: Becoming Leader. State: Replica: c540fab613cc415cab05f8f312489ff6, State: Running, Role: LEADER
I20260812 14:55:44.837045 2941 sys_catalog.cc:564] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:44.837064 2945 consensus_queue.cc:237] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540fab613cc415cab05f8f312489ff6" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44155 } }
I20260812 14:55:44.837419 2952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c540fab613cc415cab05f8f312489ff6. Latest consensus state: current_term: 1 leader_uuid: "c540fab613cc415cab05f8f312489ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540fab613cc415cab05f8f312489ff6" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44155 } } }
I20260812 14:55:44.837522 2951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c540fab613cc415cab05f8f312489ff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c540fab613cc415cab05f8f312489ff6" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 44155 } } }
I20260812 14:55:44.837570 2952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:44.837777 2951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:44.837924 2966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:44.838400 2966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:44.838853 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 2792
I20260812 14:55:44.838953 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:44.839121 4204 external_mini_cluster.cc:1428] Setting key c350a530619c82a308ecf50e5790cee4
I20260812 14:55:44.840076 2966 catalog_manager.cc:1347] Generated new cluster ID: adba9b485ad34cb1a279372642afc0e8
I20260812 14:55:44.840124 2966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:55:44 dist-test-slave-6zbq krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:44.849746 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:44.839658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.4.27.62:44155)
0812 14:55:44.839789 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:44.839797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:44.839830 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:55:44.839936 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:44.839943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:44.839992 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:44.840123 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:44.840853 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:44.842207 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:44.843278 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:44.843839 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:44.843928 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:44.846506 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:44.846510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:44.846511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:44.846522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:44.848284 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:44.848659 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:44.848662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:44.848662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:44.848754 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:44.848933 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:44.848934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:44.848934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:44.849195 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:44.849333 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:44.849420 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:55:44.849537 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260812 14:55:44.864140 2966 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:44.864571 2966 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:44.866914 2966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6: Generated new TSK 0
I20260812 14:55:44.867053 2966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:55:44.893174 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44155
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:44.989214 2984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:44.989387 2984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:44.989408 2984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:44.990481 2984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:44.990518 2984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:44.990542 2984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:44.991911 2984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:44.992072 2984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:44.992233 2984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:44.996726 2994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:44.996871 2984 server_base.cc:1034] running on GCE node
W20260812 14:55:44.996721 2999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:44.996721 2995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:44.997156 2984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:44.997373 2984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:44.998507 2984 hybrid_clock.cc:648] HybridClock initialized: now 1786546544998492 us; error 25 us; skew 500 ppm
Aug 12 14:55:44 dist-test-slave-6zbq krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546544, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:45.000845 2984 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:55:45.001237 2984 webserver.cc:460] Webserver started at http://127.4.27.1:42235/ using document root <none> and password file <none>
I20260812 14:55:45.001425 2984 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:45.001463 2984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:45.001550 2984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:45.002491 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "a41f0bf259f941099719967985e7cf75"
format_stamp: "Formatted at 2026-08-12 14:55:44 on dist-test-slave-6zbq"
server_key: "1d52ac5cd187cf6f65b0f9d45a6d4b5d"
server_key_iv: "6737899b4c4eae303e6af05990ee3339"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.002784 2984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "a41f0bf259f941099719967985e7cf75"
format_stamp: "Formatted at 2026-08-12 14:55:44 on dist-test-slave-6zbq"
server_key: "1d52ac5cd187cf6f65b0f9d45a6d4b5d"
server_key_iv: "6737899b4c4eae303e6af05990ee3339"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.003898 2984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:45.004478 3011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.004628 2984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:45.004704 2984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "a41f0bf259f941099719967985e7cf75"
format_stamp: "Formatted at 2026-08-12 14:55:44 on dist-test-slave-6zbq"
server_key: "1d52ac5cd187cf6f65b0f9d45a6d4b5d"
server_key_iv: "6737899b4c4eae303e6af05990ee3339"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.004758 2984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:45.018815 2984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:45.020496 2984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:45.020594 2984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:45.020750 2984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:45.020977 2984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:45.021006 2984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.021024 2984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:45.021040 2984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.026288 2984 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:37933
I20260812 14:55:45.026367 3198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:37933 every 8 connection(s)
I20260812 14:55:45.026619 2984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
Aug 12 14:55:45 dist-test-slave-6zbq krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546544, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:45.032823 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 2984
I20260812 14:55:45.032922 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:55:45.033120 4204 external_mini_cluster.cc:1428] Setting key 37788676fbade5454f9ad3fe70476177
I20260812 14:55:45.034958 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.027275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:46463 (local address 127.4.27.62:44155)
0812 14:55:45.027355 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.027357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.027498 (+ 141us) server_negotiation.cc:407] Connection header received
0812 14:55:45.027915 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.027917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.027926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.027972 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:45.028797 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.029277 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.030164 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.030379 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.030438 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.032266 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:45.032268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:45.032268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:45.032274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:45.033602 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.034033 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.034035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.034035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.034106 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.034329 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.034330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.034330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.034469 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:45.034528 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.034775 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.034870 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:55:45.035094 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.027233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:44155 (local address 127.4.27.1:46463)
0812 14:55:45.027387 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.027421 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.027843 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.028021 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.028024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.028542 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:45.028724 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.028729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.029339 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.029339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.030102 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.030104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.030444 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.030445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.030538 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.030950 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:45.030958 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:45.032176 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:45.033710 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:45.033719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:45.033721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:45.033961 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:45.034172 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:45.034173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:45.034173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:45.034273 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:45.034526 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:45.034533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:45.034660 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.034745 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.034888 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260812 14:55:45.035815 3204 heartbeater.cc:347] Connected to a master server at 127.4.27.62:44155
I20260812 14:55:45.035912 3204 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:45.036082 3204 heartbeater.cc:511] Master 127.4.27.62:44155 requested a full tablet report, sending...
I20260812 14:55:45.036548 2861 ts_manager.cc:194] Registered new tserver with Master: a41f0bf259f941099719967985e7cf75 (127.4.27.1:37933)
I20260812 14:55:45.037163 2861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:46463
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:55:45.057197 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44155
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:45.136428 3212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:45.136606 3212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:45.136627 3212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:45.137686 3212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:45.137732 3212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:45.137758 3212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:45.139216 3212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:45.139384 3212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:45.139549 3212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:45.144135 3231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.144134 3228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.144196 3212 server_base.cc:1034] running on GCE node
W20260812 14:55:45.144160 3229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.144515 3212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:45.144733 3212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:45.145867 3212 hybrid_clock.cc:648] HybridClock initialized: now 1786546545145852 us; error 29 us; skew 500 ppm
Aug 12 14:55:45 dist-test-slave-6zbq krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:45.148072 3212 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:55:45.148450 3212 webserver.cc:460] Webserver started at http://127.4.27.2:38705/ using document root <none> and password file <none>
I20260812 14:55:45.148643 3212 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:45.148687 3212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:45.148808 3212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:45.149757 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "19a22861bfdb48ae8dd9cca8017de25a"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "3ccb511270a59191bbaabdd57b4b8ab6"
server_key_iv: "4ed441f9303d875971a4fa75f49d7ff9"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.150074 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "19a22861bfdb48ae8dd9cca8017de25a"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "3ccb511270a59191bbaabdd57b4b8ab6"
server_key_iv: "4ed441f9303d875971a4fa75f49d7ff9"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.151412 3212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:55:45.152143 3241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.152307 3212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:45.152396 3212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "19a22861bfdb48ae8dd9cca8017de25a"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "3ccb511270a59191bbaabdd57b4b8ab6"
server_key_iv: "4ed441f9303d875971a4fa75f49d7ff9"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.152462 3212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:45.162968 3212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:45.164563 3212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:45.164693 3212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:45.164880 3212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:45.165138 3212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:45.165169 3212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.165195 3212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:45.165212 3212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.170569 3212 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:33073
I20260812 14:55:45.170634 3458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:33073 every 8 connection(s)
I20260812 14:55:45.170888 3212 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:55:45.171499 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 3212
I20260812 14:55:45.171617 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:55:45.171856 4204 external_mini_cluster.cc:1428] Setting key 16e17b385a8fbbbb918097ff5161a09c
Aug 12 14:55:45 dist-test-slave-6zbq krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
I20260812 14:55:45.179883 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.171508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:36021 (local address 127.4.27.62:44155)
0812 14:55:45.171569 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.171571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.171765 (+ 194us) server_negotiation.cc:407] Connection header received
0812 14:55:45.172338 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.172339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.172350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.172407 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:45.173492 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.173972 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.175079 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.175400 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.175448 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.177430 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:45.177431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:45.177431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:45.177438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:45.178686 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.179039 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.179040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.179041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.179096 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.179306 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.179307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.179307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.179415 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:45.179470 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.179712 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.179797 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:55:45.180028 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.171464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:44155 (local address 127.4.27.2:36021)
0812 14:55:45.171644 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.171690 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.172246 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.172472 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.172476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.173154 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:45.173407 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.173413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.174057 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.174058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.174967 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.174970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.175529 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.175530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.175632 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.176083 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:45.176091 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:45.177340 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:45.178758 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:45.178764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:45.178766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:45.178972 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:45.179150 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:45.179150 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:45.179151 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:45.179247 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:45.179469 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:45.179475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:45.179600 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.179683 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.179834 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260812 14:55:45.180445 3460 heartbeater.cc:347] Connected to a master server at 127.4.27.62:44155
I20260812 14:55:45.180521 3460 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:45.180675 3460 heartbeater.cc:511] Master 127.4.27.62:44155 requested a full tablet report, sending...
I20260812 14:55:45.181017 2861 ts_manager.cc:194] Registered new tserver with Master: 19a22861bfdb48ae8dd9cca8017de25a (127.4.27.2:33073)
I20260812 14:55:45.181367 2861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:36021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:55:45.194628 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44155
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:45.272289 3469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:45.272457 3469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:45.272477 3469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:45.273551 3469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:45.273589 3469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:45.273617 3469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:45.275027 3469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:44155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:45.275231 3469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:45.275470 3469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:45.279970 3480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.279958 3479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.279943 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.280198 3469 server_base.cc:1034] running on GCE node
I20260812 14:55:45.280364 3469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:45.280552 3469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:45.281729 3469 hybrid_clock.cc:648] HybridClock initialized: now 1786546545281721 us; error 30 us; skew 500 ppm
Aug 12 14:55:45 dist-test-slave-6zbq krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:45.284088 3469 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:55:45.284468 3469 webserver.cc:460] Webserver started at http://127.4.27.3:36757/ using document root <none> and password file <none>
I20260812 14:55:45.284682 3469 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:45.284730 3469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:45.284848 3469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:45.285791 3469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "041536ec67ce4f8598794a2d95186773"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "9947d20d9451eb3869aa45a3860214b7"
server_key_iv: "52329960c99143d58b6236d2552ed1ac"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.286115 3469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "041536ec67ce4f8598794a2d95186773"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "9947d20d9451eb3869aa45a3860214b7"
server_key_iv: "52329960c99143d58b6236d2552ed1ac"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.287344 3469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260812 14:55:45.287976 3494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.288125 3469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:45.288208 3469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "041536ec67ce4f8598794a2d95186773"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "9947d20d9451eb3869aa45a3860214b7"
server_key_iv: "52329960c99143d58b6236d2552ed1ac"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.288288 3469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:45.301159 3469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:45.302769 3469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:45.302889 3469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:45.303081 3469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:45.303341 3469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:45.303375 3469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.303399 3469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:45.303417 3469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.308995 3469 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:36761
I20260812 14:55:45.309036 3725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:36761 every 8 connection(s)
I20260812 14:55:45.309311 3469 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:55:45 dist-test-slave-6zbq krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546545, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:45.317476 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.310071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:49989 (local address 127.4.27.62:44155)
0812 14:55:45.310120 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.310122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.310419 (+ 297us) server_negotiation.cc:407] Connection header received
0812 14:55:45.310855 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.310856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.310869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.310921 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:45.311781 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.312294 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.313249 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.313453 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.313509 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.315447 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:45.315448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:45.315449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:45.315455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:45.316486 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.316808 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.316808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.316809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.316852 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.317018 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.317018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.317019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.317108 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:45.317151 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.317356 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.317413 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:55:45.317613 3731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.309945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:44155 (local address 127.4.27.3:49989)
0812 14:55:45.310113 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.310235 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.310748 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.310981 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.310985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.311506 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:45.311680 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.311685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.312379 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.312380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.313165 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.313168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.313524 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.313525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.313639 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.314048 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:45.314056 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:45.315366 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:45.316548 (+ 1182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:45.316553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:45.316555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:45.316753 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:45.316895 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:45.316895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:45.316895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:45.316979 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:45.317159 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:45.317162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:45.317264 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.317335 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.317445 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260812 14:55:45.318040 3727 heartbeater.cc:347] Connected to a master server at 127.4.27.62:44155
I20260812 14:55:45.318121 3727 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:45.318260 3727 heartbeater.cc:511] Master 127.4.27.62:44155 requested a full tablet report, sending...
I20260812 14:55:45.318667 2861 ts_manager.cc:194] Registered new tserver with Master: 041536ec67ce4f8598794a2d95186773 (127.4.27.3:36761)
I20260812 14:55:45.318704 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 3469
I20260812 14:55:45.318809 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:55:45.318990 4204 external_mini_cluster.cc:1428] Setting key b36df827be7bc11243806f89ac283e9d
I20260812 14:55:45.319075 2861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:49989
I20260812 14:55:45.319756 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:45.325825 2980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.320946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.4.27.62:44155)
0812 14:55:45.320994 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.320995 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.321074 (+ 79us) server_negotiation.cc:407] Connection header received
0812 14:55:45.321152 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.321153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.321160 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.321196 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:45.321674 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.322098 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.322787 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.322959 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.323004 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.323908 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:45.323909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:45.323909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:45.323914 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:45.325085 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.325327 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.325328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.325328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.325368 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:45.325477 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:45.325478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:45.325478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:45.325585 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:45.325653 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.325718 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.325772 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:55:45.327235 2861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:45.328723 2861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:45.334069 3748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.330522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37933 (local address 127.0.0.1:60594)
0812 14:55:45.330804 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.330842 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.330906 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.331119 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.331122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.331123 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:45.331181 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.331183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.332190 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.332192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.333016 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.333019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.333800 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.333800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.333872 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.333886 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.333955 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.334000 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":20,"thread_start_us":53,"threads_started":1}
I20260812 14:55:45.334295 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.330628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.4.27.1:37933)
0812 14:55:45.330911 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.330915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.330925 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:55:45.331013 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.331016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.331032 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.331116 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:45.331246 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.332089 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.333098 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.333721 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.333794 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.333849 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.334058 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.334099 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":79,"threads_started":1}
I20260812 14:55:45.334789 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.330522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:33073 (local address 127.0.0.1:48590)
0812 14:55:45.330967 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.330976 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.331033 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.331270 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.331271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.331274 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:45.331322 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.331324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.332279 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.332279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.333573 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.333576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.334561 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.334562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.334631 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.334634 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.334688 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.334722 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":7,"thread_start_us":48,"threads_started":1}
I20260812 14:55:45.334853 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.330635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.4.27.2:33073)
0812 14:55:45.330832 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.330836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.331053 (+ 217us) server_negotiation.cc:407] Connection header received
0812 14:55:45.331101 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.331103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.331118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.331190 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:45.331387 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.332172 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.333668 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.334416 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.334473 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.334525 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.334717 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.334774 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":76,"threads_started":1}
I20260812 14:55:45.334910 3082 tablet_service.cc:1467] Processing CreateTablet for tablet ee3992059fc643919399077194ee4a41 (DEFAULT_TABLE table=test-table [id=93f16d73e58243558744523f416803f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:45.335224 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3992059fc643919399077194ee4a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:45.335683 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.330759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.4.27.3:36761)
0812 14:55:45.330886 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.330890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.331119 (+ 229us) server_negotiation.cc:407] Connection header received
0812 14:55:45.331154 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.331156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.331171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.331245 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:45.332299 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.333345 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.334557 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.335383 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.335439 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.335472 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.335596 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.335621 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260812 14:55:45.335767 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.330703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:36761 (local address 127.0.0.1:40114)
0812 14:55:45.331018 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.331025 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.331067 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.332193 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.332193 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.332196 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:45.332226 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.332227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.333448 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.333449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.334479 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.334482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.335456 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.335457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.335519 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.335524 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.335588 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.335689 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":77,"threads_started":1}
I20260812 14:55:45.336522 3600 tablet_service.cc:1467] Processing CreateTablet for tablet ee3992059fc643919399077194ee4a41 (DEFAULT_TABLE table=test-table [id=93f16d73e58243558744523f416803f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:45.336851 3600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3992059fc643919399077194ee4a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:45.338496 3763 tablet_bootstrap.cc:492] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: Bootstrap starting.
I20260812 14:55:45.339143 3763 tablet_bootstrap.cc:654] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:45.339039 3333 tablet_service.cc:1467] Processing CreateTablet for tablet ee3992059fc643919399077194ee4a41 (DEFAULT_TABLE table=test-table [id=93f16d73e58243558744523f416803f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:45.339367 3333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3992059fc643919399077194ee4a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:45.339615 3763 log.cc:824] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:45.339880 3764 tablet_bootstrap.cc:492] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Bootstrap starting.
I20260812 14:55:45.340480 3764 tablet_bootstrap.cc:654] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:45.340971 3764 log.cc:824] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:45.341585 3764 tablet_bootstrap.cc:492] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: No bootstrap required, opened a new log
I20260812 14:55:45.341639 3764 ts_tablet_manager.cc:1397] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:45.341938 3767 tablet_bootstrap.cc:492] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Bootstrap starting.
I20260812 14:55:45.342535 3763 tablet_bootstrap.cc:492] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: No bootstrap required, opened a new log
I20260812 14:55:45.342559 3767 tablet_bootstrap.cc:654] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:45.342607 3763 ts_tablet_manager.cc:1397] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 14:55:45.342994 3767 log.cc:824] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:45.343411 3764 raft_consensus.cc:348] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.343571 3764 raft_consensus.cc:374] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:45.343605 3764 raft_consensus.cc:729] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041536ec67ce4f8598794a2d95186773, State: Initialized, Role: FOLLOWER
I20260812 14:55:45.343655 3767 tablet_bootstrap.cc:492] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: No bootstrap required, opened a new log
I20260812 14:55:45.343704 3767 ts_tablet_manager.cc:1397] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:45.343714 3764 consensus_queue.cc:260] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.343972 3727 heartbeater.cc:503] Master 127.4.27.62:44155 was elected leader, sending a full tablet report...
I20260812 14:55:45.343964 3764 ts_tablet_manager.cc:1428] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:55:45.344478 3763 raft_consensus.cc:348] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.344600 3763 raft_consensus.cc:374] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:45.344640 3763 raft_consensus.cc:729] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41f0bf259f941099719967985e7cf75, State: Initialized, Role: FOLLOWER
I20260812 14:55:45.344748 3763 consensus_queue.cc:260] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.345009 3763 ts_tablet_manager.cc:1428] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:45.345034 3204 heartbeater.cc:503] Master 127.4.27.62:44155 was elected leader, sending a full tablet report...
I20260812 14:55:45.345323 3767 raft_consensus.cc:348] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.345455 3767 raft_consensus.cc:374] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:45.345487 3767 raft_consensus.cc:729] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a22861bfdb48ae8dd9cca8017de25a, State: Initialized, Role: FOLLOWER
I20260812 14:55:45.345624 3767 consensus_queue.cc:260] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.345897 3767 ts_tablet_manager.cc:1428] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:45.345897 3460 heartbeater.cc:503] Master 127.4.27.62:44155 was elected leader, sending a full tablet report...
W20260812 14:55:45.421393 3462 tablet.cc:2369] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:45.525516 3772 raft_consensus.cc:482] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:45.525637 3772 raft_consensus.cc:504] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.526047 3772 leader_election.cc:288] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 041536ec67ce4f8598794a2d95186773 (127.4.27.3:36761), 19a22861bfdb48ae8dd9cca8017de25a (127.4.27.2:33073)
W20260812 14:55:45.527254 3205 tablet.cc:2369] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:45.529407 3761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.526286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:46295 (local address 127.4.27.3:36761)
0812 14:55:45.526360 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.526362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.526459 (+ 97us) server_negotiation.cc:407] Connection header received
0812 14:55:45.526512 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.526513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.526525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.526571 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:45.526708 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.527165 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.528499 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.529072 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.529143 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.529153 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.529302 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.529341 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:55:45.529407 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.526326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:45861 (local address 127.4.27.2:33073)
0812 14:55:45.526387 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.526388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.526396 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:55:45.526418 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.526419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.526428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.526470 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:45.526612 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.527075 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.528492 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.529072 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.529143 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.529153 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.529303 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.529339 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 14:55:45.529436 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.526196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:33073 (local address 127.4.27.1:45861)
0812 14:55:45.526253 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.526265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.526312 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.526483 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.526484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.526486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:45.526552 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.526554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.527144 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.527163 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.528401 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.528403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.529147 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.529147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.529239 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.529241 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.529286 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.529334 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":34}
I20260812 14:55:45.529443 3779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.526196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:36761 (local address 127.4.27.1:46295)
0812 14:55:45.526391 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.526401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.526459 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.526578 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.526579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.526580 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:45.526653 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.526654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.527246 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.527247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.528409 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.528410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.529147 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.529147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.529229 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.529233 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.529287 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.529334 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":49,"thread_start_us":59,"threads_started":1}
I20260812 14:55:45.529820 3377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3992059fc643919399077194ee4a41" candidate_uuid: "a41f0bf259f941099719967985e7cf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a22861bfdb48ae8dd9cca8017de25a" is_pre_election: true
I20260812 14:55:45.529814 3642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3992059fc643919399077194ee4a41" candidate_uuid: "a41f0bf259f941099719967985e7cf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041536ec67ce4f8598794a2d95186773" is_pre_election: true
I20260812 14:55:45.529958 3377 raft_consensus.cc:2454] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41f0bf259f941099719967985e7cf75 in term 0.
I20260812 14:55:45.529958 3642 raft_consensus.cc:2454] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41f0bf259f941099719967985e7cf75 in term 0.
I20260812 14:55:45.530165 3013 leader_election.cc:302] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a22861bfdb48ae8dd9cca8017de25a, a41f0bf259f941099719967985e7cf75; no voters:
I20260812 14:55:45.530329 3772 raft_consensus.cc:2781] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:45.530386 3772 raft_consensus.cc:482] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:45.530408 3772 raft_consensus.cc:3037] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:45.531073 3772 raft_consensus.cc:504] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.531196 3772 leader_election.cc:288] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [CANDIDATE]: Term 1 election: Requested vote from peers 041536ec67ce4f8598794a2d95186773 (127.4.27.3:36761), 19a22861bfdb48ae8dd9cca8017de25a (127.4.27.2:33073)
I20260812 14:55:45.531355 3642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3992059fc643919399077194ee4a41" candidate_uuid: "a41f0bf259f941099719967985e7cf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "041536ec67ce4f8598794a2d95186773"
I20260812 14:55:45.531389 3377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3992059fc643919399077194ee4a41" candidate_uuid: "a41f0bf259f941099719967985e7cf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a22861bfdb48ae8dd9cca8017de25a"
I20260812 14:55:45.531446 3377 raft_consensus.cc:3037] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:45.531440 3642 raft_consensus.cc:3037] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:45.532306 3642 raft_consensus.cc:2454] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41f0bf259f941099719967985e7cf75 in term 1.
I20260812 14:55:45.532306 3377 raft_consensus.cc:2454] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41f0bf259f941099719967985e7cf75 in term 1.
I20260812 14:55:45.532460 3013 leader_election.cc:302] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a22861bfdb48ae8dd9cca8017de25a, a41f0bf259f941099719967985e7cf75; no voters:
I20260812 14:55:45.532541 3772 raft_consensus.cc:2781] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:45.532691 3772 raft_consensus.cc:686] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 1 LEADER]: Becoming Leader. State: Replica: a41f0bf259f941099719967985e7cf75, State: Running, Role: LEADER
I20260812 14:55:45.532783 3772 consensus_queue.cc:237] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } }
I20260812 14:55:45.533514 2861 catalog_manager.cc:5495] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41f0bf259f941099719967985e7cf75 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "a41f0bf259f941099719967985e7cf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41f0bf259f941099719967985e7cf75" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260812 14:55:45.558739 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.556532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.4.27.1:37933)
0812 14:55:45.556576 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.556578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.556586 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:55:45.556646 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.556646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.556664 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.556727 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:45.556854 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.557285 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.558026 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.558206 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.558259 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.558437 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:45.558541 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:45.558612 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.558648 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.558666 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260812 14:55:45.559834 3730 tablet.cc:2369] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:45.560312 3642 raft_consensus.cc:1261] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 1 FOLLOWER]: Refusing update from remote peer a41f0bf259f941099719967985e7cf75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:45.560312 3377 raft_consensus.cc:1261] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 1 FOLLOWER]: Refusing update from remote peer a41f0bf259f941099719967985e7cf75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:45.560603 3772 consensus_queue.cc:1035] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a22861bfdb48ae8dd9cca8017de25a" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:45.560745 3780 consensus_queue.cc:1035] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "041536ec67ce4f8598794a2d95186773" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:45.562779 3790 mvcc.cc:205] Tried to move back new op lower bound from 7317694650612772864 to 7317694650502840320. Current Snapshot: MvccSnapshot[applied={T|T < 7317694650612772864}]
I20260812 14:55:45.565095 2861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:45.565172 2861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:45.565827 2861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c540fab613cc415cab05f8f312489ff6: Sending DeleteTablet for 3 replicas of tablet ee3992059fc643919399077194ee4a41
I20260812 14:55:45.566138 3333 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3992059fc643919399077194ee4a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:45 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:48590
I20260812 14:55:45.566141 3600 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3992059fc643919399077194ee4a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:45 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:40114
I20260812 14:55:45.566143 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3992059fc643919399077194ee4a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:45 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:60594
I20260812 14:55:45.566383 3805 tablet_replica.cc:330] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75: stopping tablet replica
I20260812 14:55:45.566512 3805 raft_consensus.cc:2229] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:55:45.566632 3805 raft_consensus.cc:2258] T ee3992059fc643919399077194ee4a41 P a41f0bf259f941099719967985e7cf75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:45.566677 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 2984
I20260812 14:55:45.576749 3806 tablet_replica.cc:330] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: stopping tablet replica
I20260812 14:55:45.576830 3806 raft_consensus.cc:2229] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:45.576898 3806 raft_consensus.cc:2258] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:45.577420 3806 ts_tablet_manager.cc:1905] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:45.579228 3806 ts_tablet_manager.cc:1918] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:45.579300 3806 log.cc:1196] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/ee3992059fc643919399077194ee4a41
I20260812 14:55:45.579555 3806 ts_tablet_manager.cc:1939] T ee3992059fc643919399077194ee4a41 P 19a22861bfdb48ae8dd9cca8017de25a: Deleting consensus metadata
I20260812 14:55:45.579942 2829 catalog_manager.cc:4841] TS 19a22861bfdb48ae8dd9cca8017de25a (127.4.27.2:33073): tablet ee3992059fc643919399077194ee4a41 (table test-table [id=93f16d73e58243558744523f416803f2]) successfully deleted
W20260812 14:55:45.583099 2832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:37933 (error 108)
I20260812 14:55:45.583161 3804 tablet_replica.cc:330] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: stopping tablet replica
I20260812 14:55:45.583278 3804 raft_consensus.cc:2229] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:45.583382 3804 raft_consensus.cc:2258] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:45.583443 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 3212
I20260812 14:55:45.583624 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.583520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:37933 (local address 127.0.0.1:60610)
0812 14:55:45.583566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.583596 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260812 14:55:45.583720 2832 catalog_manager.cc:4584] TS a41f0bf259f941099719967985e7cf75 (127.4.27.1:37933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee3992059fc643919399077194ee4a41: Network error: Client connection negotiation failed: client connection to 127.4.27.1:37933: connect: Connection refused (error 111)
I20260812 14:55:45.586297 3804 ts_tablet_manager.cc:1905] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:45.588116 3804 ts_tablet_manager.cc:1918] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:45.588208 3804 log.cc:1196] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/ee3992059fc643919399077194ee4a41
I20260812 14:55:45.588424 3804 ts_tablet_manager.cc:1939] T ee3992059fc643919399077194ee4a41 P 041536ec67ce4f8598794a2d95186773: Deleting consensus metadata
I20260812 14:55:45.594972 2832 catalog_manager.cc:4841] TS 041536ec67ce4f8598794a2d95186773 (127.4.27.3:36761): tablet ee3992059fc643919399077194ee4a41 (table test-table [id=93f16d73e58243558744523f416803f2]) successfully deleted
I20260812 14:55:45.597975 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 3469
I20260812 14:55:45.607328 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 2792
2026-08-12T14:55:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2203 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T14:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:45Z Disabled control of system clock
I20260812 14:55:45.638532 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:40875
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:40875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:45.716305 3812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:45.716492 3812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:45.716513 3812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:45.717612 3812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:45.717653 3812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:45.717664 3812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:45.717675 3812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:45.717695 3812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:45.719008 3812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:40875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:40875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:45.719228 3812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:45.719480 3812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:45.723822 3838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.723889 3827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.723865 3812 server_base.cc:1034] running on GCE node
W20260812 14:55:45.723834 3833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.724236 3812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:45.724469 3812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:45.725600 3812 hybrid_clock.cc:648] HybridClock initialized: now 1786546545725584 us; error 32 us; skew 500 ppm
I20260812 14:55:45.726718 3812 webserver.cc:460] Webserver started at http://127.4.27.62:41483/ using document root <none> and password file <none>
I20260812 14:55:45.726918 3812 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:45.726965 3812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:45.727064 3812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:45.728227 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "f16e3b44320643db9edd63ff5137c9ad"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "506c86383b4e05387e0edf5a1669c88d"
server_key_iv: "25e30d8bbcc0c920f7f275e5fe685ccc"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.728569 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "f16e3b44320643db9edd63ff5137c9ad"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "506c86383b4e05387e0edf5a1669c88d"
server_key_iv: "25e30d8bbcc0c920f7f275e5fe685ccc"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.729753 3812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:45.730434 3853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.730594 3812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.730686 3812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "f16e3b44320643db9edd63ff5137c9ad"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "506c86383b4e05387e0edf5a1669c88d"
server_key_iv: "25e30d8bbcc0c920f7f275e5fe685ccc"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.730754 3812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:45.738623 3812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:45.738852 3812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:45.738963 3812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:45.742499 3812 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:40875
I20260812 14:55:45.742527 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:40875 every 8 connection(s)
I20260812 14:55:45.742805 3812 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:45.743637 3947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:45.745682 3947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad: Bootstrap starting.
I20260812 14:55:45.746214 3947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:45.746502 3947 log.cc:824] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:45.747081 3947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad: No bootstrap required, opened a new log
I20260812 14:55:45.748558 3947 raft_consensus.cc:348] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e3b44320643db9edd63ff5137c9ad" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 40875 } }
I20260812 14:55:45.748683 3947 raft_consensus.cc:374] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:45.748715 3947 raft_consensus.cc:729] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16e3b44320643db9edd63ff5137c9ad, State: Initialized, Role: FOLLOWER
I20260812 14:55:45.748844 3947 consensus_queue.cc:260] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e3b44320643db9edd63ff5137c9ad" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 40875 } }
I20260812 14:55:45.748916 3947 raft_consensus.cc:388] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:45.748950 3947 raft_consensus.cc:482] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:45.748996 3947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:45.749454 3947 raft_consensus.cc:504] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e3b44320643db9edd63ff5137c9ad" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 40875 } }
I20260812 14:55:45.749540 3947 leader_election.cc:302] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f16e3b44320643db9edd63ff5137c9ad; no voters:
I20260812 14:55:45.749704 3947 leader_election.cc:288] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:45.749740 3953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:45.749874 3953 raft_consensus.cc:686] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [term 1 LEADER]: Becoming Leader. State: Replica: f16e3b44320643db9edd63ff5137c9ad, State: Running, Role: LEADER
I20260812 14:55:45.749928 3947 sys_catalog.cc:564] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:45.749971 3953 consensus_queue.cc:237] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e3b44320643db9edd63ff5137c9ad" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 40875 } }
I20260812 14:55:45.750304 3958 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f16e3b44320643db9edd63ff5137c9ad. Latest consensus state: current_term: 1 leader_uuid: "f16e3b44320643db9edd63ff5137c9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e3b44320643db9edd63ff5137c9ad" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 40875 } } }
I20260812 14:55:45.750401 3958 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:45.750447 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f16e3b44320643db9edd63ff5137c9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16e3b44320643db9edd63ff5137c9ad" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 40875 } } }
I20260812 14:55:45.750496 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:45.750617 3967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:45.751082 3967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:45.752377 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 3812
I20260812 14:55:45.752471 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:45.752502 3967 catalog_manager.cc:1347] Generated new cluster ID: 2c3d1b9db8e446e4b985b932ef52fef3
I20260812 14:55:45.752552 3967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:45.752663 4204 external_mini_cluster.cc:1428] Setting key 7a46ac1211642f125424f5703c43e2a7
I20260812 14:55:45.757138 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.753153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.4.27.62:40875)
0812 14:55:45.753318 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.753324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.753346 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:55:45.753428 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.753434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.753488 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.753595 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:45.753792 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.754742 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.755473 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.755900 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.756004 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.756588 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:45.756592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:45.756592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:45.756750 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:45.756781 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:45.756791 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:45.756843 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.756914 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.756985 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260812 14:55:45.765293 3967 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:45.765914 3967 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:45.779548 3967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad: Generated new TSK 0
I20260812 14:55:45.779722 3967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:45.781531 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:40875
--builtin_ntp_servers=127.4.27.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:45.863243 3994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:45.863425 3994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:45.863451 3994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:45.864524 3994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:45.864562 3994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:45.864593 3994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:45.865885 3994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:45.866115 3994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:45.866407 3994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:45.870980 4003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.870985 4002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.870985 4006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.871186 3994 server_base.cc:1034] running on GCE node
I20260812 14:55:45.871404 3994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:45.871667 3994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:45.872821 3994 hybrid_clock.cc:648] HybridClock initialized: now 1786546545872802 us; error 31 us; skew 500 ppm
I20260812 14:55:45.874193 3994 webserver.cc:460] Webserver started at http://127.4.27.1:38391/ using document root <none> and password file <none>
I20260812 14:55:45.874508 3994 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:45.874569 3994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:45.874686 3994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:45.875898 3994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "dfde435efdaa4e89aec94366287ce071"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "035e7a567fbc4e3473d6b364b2db936a"
server_key_iv: "64ed590c647868ff443c81c338e2e00b"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.876266 3994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "dfde435efdaa4e89aec94366287ce071"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "035e7a567fbc4e3473d6b364b2db936a"
server_key_iv: "64ed590c647868ff443c81c338e2e00b"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.877710 3994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:55:45.878445 4015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.878592 3994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:45.878674 3994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "dfde435efdaa4e89aec94366287ce071"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "035e7a567fbc4e3473d6b364b2db936a"
server_key_iv: "64ed590c647868ff443c81c338e2e00b"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.878744 3994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:45.893445 3994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:45.893687 3994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:45.893800 3994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:45.893977 3994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:45.894254 3994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:45.894285 3994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.894304 3994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:45.894327 3994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:45.899471 3994 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:45115
I20260812 14:55:45.899549 4228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:45115 every 8 connection(s)
I20260812 14:55:45.899794 3994 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:45.904100 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.900385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:58765 (local address 127.4.27.62:40875)
0812 14:55:45.900430 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:45.900432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:45.900642 (+ 210us) server_negotiation.cc:407] Connection header received
0812 14:55:45.901051 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:45.901052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:45.901063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:45.901119 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:45.901699 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.902166 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.903074 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.903257 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.903299 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.903908 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:45.903909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:45.903909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:45.903914 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:45.903929 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:45.903933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:45.903970 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:45.904021 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:55:45.904043 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:55:45.904299 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:45.900364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:40875 (local address 127.4.27.1:58765)
0812 14:55:45.900525 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:45.900556 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:45.900977 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:45.901176 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:45.901179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:45.901187 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:45.901636 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.901640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.902226 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.902227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.903011 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:45.903014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:45.903311 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:45.903311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:45.903412 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:45.903795 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:45.903795 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:45.903819 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:45.903840 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:45.903971 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:45.903973 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:45.903975 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:55:45.904018 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:55:45.904089 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260812 14:55:45.904735 4231 heartbeater.cc:347] Connected to a master server at 127.4.27.62:40875
I20260812 14:55:45.904819 4231 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:45.904979 4231 heartbeater.cc:511] Master 127.4.27.62:40875 requested a full tablet report, sending...
I20260812 14:55:45.905402 3882 ts_manager.cc:194] Registered new tserver with Master: dfde435efdaa4e89aec94366287ce071 (127.4.27.1:45115)
I20260812 14:55:45.905967 3882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.1:58765
I20260812 14:55:45.907375 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 3994
I20260812 14:55:45.907459 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:55:45.907636 4204 external_mini_cluster.cc:1428] Setting key 2974507c5596641e59fc994e98f1b940
I20260812 14:55:45.908464 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:40875
--builtin_ntp_servers=127.4.27.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:45.986956 4235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:45.987126 4235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:45.987146 4235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:45.988214 4235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:45.988258 4235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:45.988283 4235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:45.989609 4235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:45.989812 4235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:45.990062 4235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:45.994571 4244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:45.994577 4247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.994591 4235 server_base.cc:1034] running on GCE node
W20260812 14:55:45.994577 4249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:45.994925 4235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:45.995124 4235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:45.996299 4235 hybrid_clock.cc:648] HybridClock initialized: now 1786546545996285 us; error 32 us; skew 500 ppm
I20260812 14:55:45.997443 4235 webserver.cc:460] Webserver started at http://127.4.27.2:41227/ using document root <none> and password file <none>
I20260812 14:55:45.997649 4235 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:45.997692 4235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:45.997810 4235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:45.998797 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "96ef4877212040728a0cfef98b0f81d3"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "66aa5870a3bdeb564ebb091deb87b07e"
server_key_iv: "0566adb0691a3e3d6809f5f97a23a803"
server_key_version: "encryptionkey@0"
I20260812 14:55:45.999104 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "96ef4877212040728a0cfef98b0f81d3"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "66aa5870a3bdeb564ebb091deb87b07e"
server_key_iv: "0566adb0691a3e3d6809f5f97a23a803"
server_key_version: "encryptionkey@0"
I20260812 14:55:46.000257 4235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:46.000857 4258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:46.000986 4235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:46.001066 4235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "96ef4877212040728a0cfef98b0f81d3"
format_stamp: "Formatted at 2026-08-12 14:55:45 on dist-test-slave-6zbq"
server_key: "66aa5870a3bdeb564ebb091deb87b07e"
server_key_iv: "0566adb0691a3e3d6809f5f97a23a803"
server_key_version: "encryptionkey@0"
I20260812 14:55:46.001134 4235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:46.015259 4235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:46.015483 4235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:46.015599 4235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:46.015781 4235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:46.016050 4235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:46.016083 4235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:46.016108 4235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:46.016125 4235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:46.021220 4235 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:35429
I20260812 14:55:46.021291 4468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:35429 every 8 connection(s)
I20260812 14:55:46.021530 4235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:55:46.023195 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4235
I20260812 14:55:46.023279 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:55:46.023468 4204 external_mini_cluster.cc:1428] Setting key 4c80725a8997c17c64912337c1ad9a54
I20260812 14:55:46.024308 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:40875
--builtin_ntp_servers=127.4.27.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:55:46.026481 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.022147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:50769 (local address 127.4.27.62:40875)
0812 14:55:46.022207 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.022208 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.022417 (+ 209us) server_negotiation.cc:407] Connection header received
0812 14:55:46.022878 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.022880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.022896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.022945 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:46.023604 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.024195 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.025226 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.025437 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.025493 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.026234 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:46.026236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:46.026236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:46.026241 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:46.026258 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:46.026262 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:46.026303 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.026394 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.026425 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:55:46.026690 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.022109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:40875 (local address 127.4.27.2:50769)
0812 14:55:46.022299 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.022328 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.022789 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.023008 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.023012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.023041 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:46.023528 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.023534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.024275 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.024277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.025146 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.025149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.025510 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.025511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.025650 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.026096 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:46.026097 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:46.026132 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:46.026159 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:46.026306 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:46.026310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:46.026313 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.026396 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.026493 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260812 14:55:46.027184 4472 heartbeater.cc:347] Connected to a master server at 127.4.27.62:40875
I20260812 14:55:46.027277 4472 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:46.027442 4472 heartbeater.cc:511] Master 127.4.27.62:40875 requested a full tablet report, sending...
I20260812 14:55:46.027753 3882 ts_manager.cc:194] Registered new tserver with Master: 96ef4877212040728a0cfef98b0f81d3 (127.4.27.2:35429)
I20260812 14:55:46.028096 3882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.2:50769
W20260812 14:55:46.105321 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:46.105494 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:46.105513 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:46.106634 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:46.106671 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:46.106696 4475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:46.108000 4475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:40875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:46.108196 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:46.108448 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:55:46.112987 4475 server_base.cc:1034] running on GCE node
W20260812 14:55:46.112946 4489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:46.113039 4493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:46.112958 4490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:46.113399 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:46.113631 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:46.114763 4475 hybrid_clock.cc:648] HybridClock initialized: now 1786546546114744 us; error 30 us; skew 500 ppm
I20260812 14:55:46.115876 4475 webserver.cc:460] Webserver started at http://127.4.27.3:38871/ using document root <none> and password file <none>
I20260812 14:55:46.116097 4475 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:46.116143 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:46.116282 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:46.117239 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "bc6be0229da5477ca749e6371847b2bb"
format_stamp: "Formatted at 2026-08-12 14:55:46 on dist-test-slave-6zbq"
server_key: "f0d538623c57fa40847cc20dd2f185b1"
server_key_iv: "1b4d5f0e651fb75da1fc07f836c30a61"
server_key_version: "encryptionkey@0"
I20260812 14:55:46.117556 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "bc6be0229da5477ca749e6371847b2bb"
format_stamp: "Formatted at 2026-08-12 14:55:46 on dist-test-slave-6zbq"
server_key: "f0d538623c57fa40847cc20dd2f185b1"
server_key_iv: "1b4d5f0e651fb75da1fc07f836c30a61"
server_key_version: "encryptionkey@0"
I20260812 14:55:46.118780 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:55:46.119407 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:46.119575 4475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:46.119647 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "bc6be0229da5477ca749e6371847b2bb"
format_stamp: "Formatted at 2026-08-12 14:55:46 on dist-test-slave-6zbq"
server_key: "f0d538623c57fa40847cc20dd2f185b1"
server_key_iv: "1b4d5f0e651fb75da1fc07f836c30a61"
server_key_version: "encryptionkey@0"
I20260812 14:55:46.119719 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:46.129249 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:46.129469 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:46.129581 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:46.129776 4475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:46.130043 4475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:46.130077 4475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:46.130102 4475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:46.130119 4475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:46.135494 4475 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:42351
I20260812 14:55:46.135528 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:42351 every 8 connection(s)
I20260812 14:55:46.135799 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:55:46.139912 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4475
I20260812 14:55:46.140021 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:55:46.140231 4204 external_mini_cluster.cc:1428] Setting key daff1248167dd06aae56e827f8dbaf9b
I20260812 14:55:46.140425 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.136418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:50373 (local address 127.4.27.62:40875)
0812 14:55:46.136459 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.136461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.136655 (+ 194us) server_negotiation.cc:407] Connection header received
0812 14:55:46.137083 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.137085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.137097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.137150 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:46.137767 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.138244 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.139166 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.139354 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.139409 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.140152 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:46.140154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:46.140154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:46.140159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:46.140176 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:46.140181 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:46.140231 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.140319 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.140347 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:55:46.140600 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.136392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:40875 (local address 127.4.27.3:50373)
0812 14:55:46.136558 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.136586 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.136994 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.137222 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.137226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.137236 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:46.137684 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.137688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.138325 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.138326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.139084 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.139086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.139421 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.139422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.139546 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.139992 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:46.139993 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:46.140026 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:46.140053 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:46.140243 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:46.140246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:46.140249 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.140300 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.140396 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260812 14:55:46.140986 4706 heartbeater.cc:347] Connected to a master server at 127.4.27.62:40875
I20260812 14:55:46.141072 4706 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:46.141249 4706 heartbeater.cc:511] Master 127.4.27.62:40875 requested a full tablet report, sending...
I20260812 14:55:46.141569 3882 ts_manager.cc:194] Registered new tserver with Master: bc6be0229da5477ca749e6371847b2bb (127.4.27.3:42351)
I20260812 14:55:46.141901 3882 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.3:50373
I20260812 14:55:46.151777 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:46.155323 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.152986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.4.27.62:40875)
0812 14:55:46.153050 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.153052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.153110 (+ 58us) server_negotiation.cc:407] Connection header received
0812 14:55:46.153165 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.153166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.153174 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.153226 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:46.153340 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.153773 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.154440 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.154610 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.154657 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.155159 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:46.155159 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:46.155159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:46.155163 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:46.155172 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:46.155175 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:46.155208 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.155245 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.155260 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:55:46.156690 3882 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:46.158093 3882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:46.163818 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.159983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35429 (local address 127.0.0.1:42676)
0812 14:55:46.160275 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.160304 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.160370 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.160745 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.160745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.160748 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:46.160777 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.160778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.161661 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.161662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.162872 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.162875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.163528 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.163606 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.163627 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.163691 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.163741 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":82,"threads_started":1}
I20260812 14:55:46.163822 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.160089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.4.27.2:35429)
0812 14:55:46.160245 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.160249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.160447 (+ 198us) server_negotiation.cc:407] Connection header received
0812 14:55:46.160487 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.160489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.160503 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.160578 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:46.160848 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.161575 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.162963 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.163447 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163507 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.163590 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.163709 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.163748 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260812 14:55:46.163975 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.160152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:45115 (local address 127.0.0.1:41656)
0812 14:55:46.160346 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.160355 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.160394 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.160690 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.160691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.160693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:46.160734 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.160736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.161965 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.161966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.162790 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.162792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.163759 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.163818 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.163822 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.163867 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.163903 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":59,"threads_started":1}
I20260812 14:55:46.164086 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.160027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.4.27.3:42351)
0812 14:55:46.160166 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.160170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.160356 (+ 186us) server_negotiation.cc:407] Connection header received
0812 14:55:46.160485 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.160488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.160503 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.160578 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:46.160757 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.161822 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163022 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.163666 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163732 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.163801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.163980 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.164018 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260812 14:55:46.164160 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.160347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.4.27.1:45115)
0812 14:55:46.160530 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.160533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.160540 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:55:46.160579 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.160581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.160596 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.160675 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:46.160822 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.161903 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.162873 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.163571 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163637 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.163721 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.164042 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.164083 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260812 14:55:46.164053 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.159968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:42351 (local address 127.0.0.1:56848)
0812 14:55:46.160264 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.160296 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.160384 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.160595 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.160598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.160600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:46.160669 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.160670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.161905 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.161906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.162953 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.162956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.163830 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.163830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.163881 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.163884 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.163954 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.163984 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":48,"threads_started":1}
I20260812 14:55:46.164825 4598 tablet_service.cc:1467] Processing CreateTablet for tablet 51a5e956ef0843199fdd29064eb0a4fd (DEFAULT_TABLE table=test-table [id=8568bb70ef024ab4a9a6e748cf5e34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:46.164801 4338 tablet_service.cc:1467] Processing CreateTablet for tablet 51a5e956ef0843199fdd29064eb0a4fd (DEFAULT_TABLE table=test-table [id=8568bb70ef024ab4a9a6e748cf5e34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:46.164855 4106 tablet_service.cc:1467] Processing CreateTablet for tablet 51a5e956ef0843199fdd29064eb0a4fd (DEFAULT_TABLE table=test-table [id=8568bb70ef024ab4a9a6e748cf5e34c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:46.165136 4338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a5e956ef0843199fdd29064eb0a4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:46.165136 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a5e956ef0843199fdd29064eb0a4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:46.165153 4106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a5e956ef0843199fdd29064eb0a4fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:46.167500 4727 tablet_bootstrap.cc:492] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: Bootstrap starting.
I20260812 14:55:46.168078 4727 tablet_bootstrap.cc:654] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:46.168176 4728 tablet_bootstrap.cc:492] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Bootstrap starting.
I20260812 14:55:46.168485 4727 log.cc:824] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:46.168789 4728 tablet_bootstrap.cc:654] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:46.169054 4728 log.cc:824] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:46.169234 4727 tablet_bootstrap.cc:492] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: No bootstrap required, opened a new log
I20260812 14:55:46.169304 4727 ts_tablet_manager.cc:1397] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:46.169394 4730 tablet_bootstrap.cc:492] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Bootstrap starting.
I20260812 14:55:46.169685 4728 tablet_bootstrap.cc:492] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: No bootstrap required, opened a new log
I20260812 14:55:46.169745 4728 ts_tablet_manager.cc:1397] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:46.169968 4730 tablet_bootstrap.cc:654] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:46.170329 4730 log.cc:824] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:46.171113 4730 tablet_bootstrap.cc:492] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: No bootstrap required, opened a new log
I20260812 14:55:46.171237 4730 ts_tablet_manager.cc:1397] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 14:55:46.171643 4727 raft_consensus.cc:348] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.171808 4727 raft_consensus.cc:374] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:46.171844 4727 raft_consensus.cc:729] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfde435efdaa4e89aec94366287ce071, State: Initialized, Role: FOLLOWER
I20260812 14:55:46.171816 4728 raft_consensus.cc:348] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.171972 4728 raft_consensus.cc:374] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:46.172008 4728 raft_consensus.cc:729] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ef4877212040728a0cfef98b0f81d3, State: Initialized, Role: FOLLOWER
I20260812 14:55:46.171995 4727 consensus_queue.cc:260] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.172142 4728 consensus_queue.cc:260] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.172261 4727 ts_tablet_manager.cc:1428] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:55:46.172394 4728 ts_tablet_manager.cc:1428] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 14:55:46.172402 4231 heartbeater.cc:503] Master 127.4.27.62:40875 was elected leader, sending a full tablet report...
I20260812 14:55:46.172508 4472 heartbeater.cc:503] Master 127.4.27.62:40875 was elected leader, sending a full tablet report...
I20260812 14:55:46.173518 4730 raft_consensus.cc:348] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.173655 4730 raft_consensus.cc:374] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:46.173688 4730 raft_consensus.cc:729] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6be0229da5477ca749e6371847b2bb, State: Initialized, Role: FOLLOWER
I20260812 14:55:46.173786 4730 consensus_queue.cc:260] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.173982 4730 ts_tablet_manager.cc:1428] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:55:46.174041 4706 heartbeater.cc:503] Master 127.4.27.62:40875 was elected leader, sending a full tablet report...
W20260812 14:55:46.272039 4473 tablet.cc:2369] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:55:46.386323 4707 tablet.cc:2369] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:55:46.400409 4233 tablet.cc:2369] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:46.502797 4743 raft_consensus.cc:482] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:46.502910 4743 raft_consensus.cc:504] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.503342 4743 leader_election.cc:288] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96ef4877212040728a0cfef98b0f81d3 (127.4.27.2:35429), dfde435efdaa4e89aec94366287ce071 (127.4.27.1:45115)
I20260812 14:55:46.506044 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.503556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:43277 (local address 127.4.27.2:35429)
0812 14:55:46.503678 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.503680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.503691 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:55:46.503721 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.503722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.503732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.503799 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:46.503965 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.504509 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.505382 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.505742 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.505804 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.505815 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.505946 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.505981 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 14:55:46.506070 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.503474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:35429 (local address 127.4.27.3:43277)
0812 14:55:46.503530 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.503543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.503597 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.503808 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.503809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.503811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:46.503880 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.503882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.504577 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.504578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.505298 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.505299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.505812 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.505813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.505882 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.505885 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.505931 (+ 46us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.505972 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:55:46.506085 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.503663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:55887 (local address 127.4.27.1:45115)
0812 14:55:46.503710 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.503712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.503799 (+ 87us) server_negotiation.cc:407] Connection header received
0812 14:55:46.503832 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.503833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.503844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.503910 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:46.504016 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.504480 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.505415 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.505763 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.505818 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.505828 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.505983 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.506022 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:55:46.506131 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.503577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:45115 (local address 127.4.27.3:55887)
0812 14:55:46.503729 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.503737 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:46.503788 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:46.503902 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:46.503903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:46.503904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:46.503959 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.503961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.504545 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.504546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.505332 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:46.505335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.505845 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.505846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:46.505913 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.505916 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:46.505967 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:55:46.506003 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260812 14:55:46.506410 4138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a5e956ef0843199fdd29064eb0a4fd" candidate_uuid: "bc6be0229da5477ca749e6371847b2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfde435efdaa4e89aec94366287ce071" is_pre_election: true
I20260812 14:55:46.506415 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a5e956ef0843199fdd29064eb0a4fd" candidate_uuid: "bc6be0229da5477ca749e6371847b2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ef4877212040728a0cfef98b0f81d3" is_pre_election: true
I20260812 14:55:46.506563 4374 raft_consensus.cc:2454] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6be0229da5477ca749e6371847b2bb in term 0.
I20260812 14:55:46.506563 4138 raft_consensus.cc:2454] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc6be0229da5477ca749e6371847b2bb in term 0.
I20260812 14:55:46.506798 4505 leader_election.cc:302] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96ef4877212040728a0cfef98b0f81d3, bc6be0229da5477ca749e6371847b2bb; no voters:
I20260812 14:55:46.506913 4743 raft_consensus.cc:2781] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:46.506969 4743 raft_consensus.cc:482] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:46.506999 4743 raft_consensus.cc:3037] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:46.507776 4743 raft_consensus.cc:504] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.507910 4743 leader_election.cc:288] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [CANDIDATE]: Term 1 election: Requested vote from peers 96ef4877212040728a0cfef98b0f81d3 (127.4.27.2:35429), dfde435efdaa4e89aec94366287ce071 (127.4.27.1:45115)
I20260812 14:55:46.508068 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a5e956ef0843199fdd29064eb0a4fd" candidate_uuid: "bc6be0229da5477ca749e6371847b2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ef4877212040728a0cfef98b0f81d3"
I20260812 14:55:46.508150 4374 raft_consensus.cc:3037] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:46.508162 4138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a5e956ef0843199fdd29064eb0a4fd" candidate_uuid: "bc6be0229da5477ca749e6371847b2bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfde435efdaa4e89aec94366287ce071"
I20260812 14:55:46.508235 4138 raft_consensus.cc:3037] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:46.509054 4138 raft_consensus.cc:2454] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6be0229da5477ca749e6371847b2bb in term 1.
I20260812 14:55:46.509054 4374 raft_consensus.cc:2454] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc6be0229da5477ca749e6371847b2bb in term 1.
I20260812 14:55:46.509225 4505 leader_election.cc:302] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96ef4877212040728a0cfef98b0f81d3, bc6be0229da5477ca749e6371847b2bb; no voters:
I20260812 14:55:46.509312 4743 raft_consensus.cc:2781] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:46.509439 4743 raft_consensus.cc:686] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 1 LEADER]: Becoming Leader. State: Replica: bc6be0229da5477ca749e6371847b2bb, State: Running, Role: LEADER
I20260812 14:55:46.509534 4743 consensus_queue.cc:237] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } }
I20260812 14:55:46.510362 3878 catalog_manager.cc:5495] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb reported cstate change: term changed from 0 to 1, leader changed from <none> to bc6be0229da5477ca749e6371847b2bb (127.4.27.3). New cstate: current_term: 1 leader_uuid: "bc6be0229da5477ca749e6371847b2bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc6be0229da5477ca749e6371847b2bb" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 42351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 } health_report { overall_health: UNKNOWN } } }
I20260812 14:55:46.515465 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.513637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.4.27.3:42351)
0812 14:55:46.513673 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:46.513674 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:46.513682 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:55:46.513720 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:46.513721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:46.513734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:46.513789 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:46.513873 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.514284 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.514934 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:46.515102 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:46.515146 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:46.515237 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:46.515318 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:46.515378 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:46.515393 (+ 15us) server_negotiation.cc:299] Negotiation successful
0812 14:55:46.515405 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:55:46.516956 4374 raft_consensus.cc:1261] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 1 FOLLOWER]: Refusing update from remote peer bc6be0229da5477ca749e6371847b2bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:46.516959 4138 raft_consensus.cc:1261] T 51a5e956ef0843199fdd29064eb0a4fd P dfde435efdaa4e89aec94366287ce071 [term 1 FOLLOWER]: Refusing update from remote peer bc6be0229da5477ca749e6371847b2bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:46.517257 4747 consensus_queue.cc:1035] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ef4877212040728a0cfef98b0f81d3" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 35429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:46.517365 4743 consensus_queue.cc:1035] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfde435efdaa4e89aec94366287ce071" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:46.519497 4751 mvcc.cc:205] Tried to move back new op lower bound from 7317694654531248128 to 7317694654503469056. Current Snapshot: MvccSnapshot[applied={T|T < 7317694654531248128}]
I20260812 14:55:46.519656 4752 mvcc.cc:205] Tried to move back new op lower bound from 7317694654531248128 to 7317694654503469056. Current Snapshot: MvccSnapshot[applied={T|T < 7317694654531248128}]
I20260812 14:55:46.521803 3882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:46.521889 3882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:46.522589 3882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f16e3b44320643db9edd63ff5137c9ad: Sending DeleteTablet for 3 replicas of tablet 51a5e956ef0843199fdd29064eb0a4fd
I20260812 14:55:46.522881 4598 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a5e956ef0843199fdd29064eb0a4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:46 UTC) from {username='slave'} at 127.0.0.1:56848
I20260812 14:55:46.522941 4338 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a5e956ef0843199fdd29064eb0a4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:46 UTC) from {username='slave'} at 127.0.0.1:42676
I20260812 14:55:46.523147 4771 tablet_replica.cc:330] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: stopping tablet replica
I20260812 14:55:46.523138 4106 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a5e956ef0843199fdd29064eb0a4fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:46 UTC) from {username='slave'} at 127.0.0.1:41656
I20260812 14:55:46.523252 4771 raft_consensus.cc:2229] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:46.523351 4771 raft_consensus.cc:2258] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:46.523574 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 3994
I20260812 14:55:46.523736 4771 ts_tablet_manager.cc:1905] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:46.524930 4771 ts_tablet_manager.cc:1918] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:46.525041 4771 log.cc:1196] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/51a5e956ef0843199fdd29064eb0a4fd
I20260812 14:55:46.525301 4771 ts_tablet_manager.cc:1939] T 51a5e956ef0843199fdd29064eb0a4fd P 96ef4877212040728a0cfef98b0f81d3: Deleting consensus metadata
I20260812 14:55:46.525645 3854 catalog_manager.cc:4841] TS 96ef4877212040728a0cfef98b0f81d3 (127.4.27.2:35429): tablet 51a5e956ef0843199fdd29064eb0a4fd (table test-table [id=8568bb70ef024ab4a9a6e748cf5e34c5]) successfully deleted
I20260812 14:55:46.535750 4769 tablet_replica.cc:330] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: stopping tablet replica
I20260812 14:55:46.535857 4769 raft_consensus.cc:2229] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:55:46.535983 4769 raft_consensus.cc:2258] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:46.536512 4769 ts_tablet_manager.cc:1905] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:46.537992 4769 ts_tablet_manager.cc:1918] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:46.538064 4769 log.cc:1196] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/51a5e956ef0843199fdd29064eb0a4fd
I20260812 14:55:46.538305 4769 ts_tablet_manager.cc:1939] T 51a5e956ef0843199fdd29064eb0a4fd P bc6be0229da5477ca749e6371847b2bb: Deleting consensus metadata
I20260812 14:55:46.539268 3858 catalog_manager.cc:4841] TS bc6be0229da5477ca749e6371847b2bb (127.4.27.3:42351): tablet 51a5e956ef0843199fdd29064eb0a4fd (table test-table [id=8568bb70ef024ab4a9a6e748cf5e34c5]) successfully deleted
W20260812 14:55:46.540670 3854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:45115 (error 108)
I20260812 14:55:46.541134 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4235
I20260812 14:55:46.541355 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:46.541242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:45115 (local address 127.0.0.1:41672)
0812 14:55:46.541290 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:46.541327 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:45115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 14:55:46.541447 3854 catalog_manager.cc:4584] TS dfde435efdaa4e89aec94366287ce071 (127.4.27.1:45115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51a5e956ef0843199fdd29064eb0a4fd: Network error: Client connection negotiation failed: client connection to 127.4.27.1:45115: connect: Connection refused (error 111)
I20260812 14:55:46.553508 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4475
I20260812 14:55:46.563997 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 3812
2026-08-12T14:55:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (954 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:46 dist-test-slave-6zbq krb5kdc[4780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:46 dist-test-slave-6zbq krb5kdc[4780](info): set up 2 sockets
Aug 12 14:55:46 dist-test-slave-6zbq krb5kdc[4780](info): commencing operation
krb5kdc: starting...
W20260812 14:55:47.750924 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:47 dist-test-slave-6zbq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:55:47.862180 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:43909
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:43909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:47.940874 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:47.941056 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:47.941076 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:47.942198 4803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:47.942240 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:47.942253 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:47.942265 4803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:47.942276 4803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:47.944090 4803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:43909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:47.944308 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:47.944520 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:47.949033 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:47.949035 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:47.949026 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:47.949172 4803 server_base.cc:1034] running on GCE node
I20260812 14:55:47.949496 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:47.949744 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:47.950875 4803 hybrid_clock.cc:648] HybridClock initialized: now 1786546547950863 us; error 30 us; skew 500 ppm
Aug 12 14:55:47 dist-test-slave-6zbq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546547, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:47.957127 4803 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:55:47.957521 4803 webserver.cc:460] Webserver started at http://127.4.27.62:44159/ using document root <none> and password file <none>
I20260812 14:55:47.957729 4803 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:47.957777 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:47.957901 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:47.958933 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "10af38ed5abc4ae3a5148b2e4802267c"
format_stamp: "Formatted at 2026-08-12 14:55:47 on dist-test-slave-6zbq"
server_key: "95b754cfeb09f295d2cb11b37950c7c0"
server_key_iv: "08088ec4c7f292019b8317e427f97ed5"
server_key_version: "encryptionkey@0"
I20260812 14:55:47.959257 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "10af38ed5abc4ae3a5148b2e4802267c"
format_stamp: "Formatted at 2026-08-12 14:55:47 on dist-test-slave-6zbq"
server_key: "95b754cfeb09f295d2cb11b37950c7c0"
server_key_iv: "08088ec4c7f292019b8317e427f97ed5"
server_key_version: "encryptionkey@0"
I20260812 14:55:47.960433 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:47.961043 4826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:47.961207 4803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:47.961293 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "10af38ed5abc4ae3a5148b2e4802267c"
format_stamp: "Formatted at 2026-08-12 14:55:47 on dist-test-slave-6zbq"
server_key: "95b754cfeb09f295d2cb11b37950c7c0"
server_key_iv: "08088ec4c7f292019b8317e427f97ed5"
server_key_version: "encryptionkey@0"
I20260812 14:55:47.961354 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:47.979143 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:47.980880 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:47.981006 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:47.984719 4803 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:43909
I20260812 14:55:47.984750 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:43909 every 8 connection(s)
I20260812 14:55:47.985038 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:47.985857 4935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:47.986232 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4803
I20260812 14:55:47.986377 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:47.986595 4204 external_mini_cluster.cc:1428] Setting key bf9d7ee5c123d8bff8e13b99537aedea
I20260812 14:55:47.988687 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c: Bootstrap starting.
I20260812 14:55:47.989333 4935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:47.989624 4935 log.cc:824] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:47.990305 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c: No bootstrap required, opened a new log
I20260812 14:55:47.992425 4935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af38ed5abc4ae3a5148b2e4802267c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43909 } }
I20260812 14:55:47.992553 4935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:47.992588 4935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10af38ed5abc4ae3a5148b2e4802267c, State: Initialized, Role: FOLLOWER
I20260812 14:55:47.992709 4935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af38ed5abc4ae3a5148b2e4802267c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43909 } }
I20260812 14:55:47.992787 4935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:47.992820 4935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:47.992866 4935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 0 FOLLOWER]: Advancing to term 1
Aug 12 14:55:47 dist-test-slave-6zbq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:47.993371 4935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af38ed5abc4ae3a5148b2e4802267c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43909 } }
I20260812 14:55:47.993474 4935 leader_election.cc:302] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10af38ed5abc4ae3a5148b2e4802267c; no voters:
I20260812 14:55:47.993637 4935 leader_election.cc:288] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:47.993675 4946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:47.993867 4946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [term 1 LEADER]: Becoming Leader. State: Replica: 10af38ed5abc4ae3a5148b2e4802267c, State: Running, Role: LEADER
I20260812 14:55:47.993952 4935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:47.994015 4946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af38ed5abc4ae3a5148b2e4802267c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43909 } }
I20260812 14:55:47.994505 4949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10af38ed5abc4ae3a5148b2e4802267c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af38ed5abc4ae3a5148b2e4802267c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43909 } } }
I20260812 14:55:47.994589 4949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:47.994741 4950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10af38ed5abc4ae3a5148b2e4802267c. Latest consensus state: current_term: 1 leader_uuid: "10af38ed5abc4ae3a5148b2e4802267c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af38ed5abc4ae3a5148b2e4802267c" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 43909 } } }
I20260812 14:55:47.994781 4963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:47.994843 4950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:47.995352 4963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:47.996949 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:47.987039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.4.27.62:43909)
0812 14:55:47.987232 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:47.987241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:47.987275 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:55:47.987758 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:47.987767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:47.987824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:47.987969 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:47.988514 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:47.989788 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:47.990603 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:47.991193 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:47.991314 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:47.993421 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:47.993426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:47.993426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:47.993438 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:47.995358 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:47.995752 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:47.995755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:47.995755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:47.995850 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:47.996053 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:47.996054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:47.996055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:47.996339 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:47.996470 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:47.996628 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 14:55:47.996772 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260812 14:55:47.996989 4963 catalog_manager.cc:1347] Generated new cluster ID: fc14026b978443d8ab7b3b3a11129af6
I20260812 14:55:47.997031 4963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:48.016103 4963 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:48.016768 4963 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:48.023730 4963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c: Generated new TSK 0
I20260812 14:55:48.023885 4963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:55:48.052076 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:43909
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:48.130801 4986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:48.130972 4986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:48.130993 4986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:48.132068 4986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:48.132108 4986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:48.132136 4986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:48.133543 4986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:48.133752 4986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:48.133956 4986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:48.138484 4997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.138463 4999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.138466 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:48.138559 4986 server_base.cc:1034] running on GCE node
I20260812 14:55:48.138881 4986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:48.139096 4986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:48.140228 4986 hybrid_clock.cc:648] HybridClock initialized: now 1786546548140217 us; error 24 us; skew 500 ppm
Aug 12 14:55:48 dist-test-slave-6zbq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:48.142519 4986 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:55:48.142903 4986 webserver.cc:460] Webserver started at http://127.4.27.1:35959/ using document root <none> and password file <none>
I20260812 14:55:48.143115 4986 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:48.143162 4986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:48.143285 4986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:48.144193 4986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "6e5ec00cfc094015a6a59ee2238fe4a5"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "0a945928ec663271e1441a5c9022f5d0"
server_key_iv: "bfc405a3192136c5629193717e83bec1"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.144505 4986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "6e5ec00cfc094015a6a59ee2238fe4a5"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "0a945928ec663271e1441a5c9022f5d0"
server_key_iv: "bfc405a3192136c5629193717e83bec1"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.145711 4986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:55:48.146436 5010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.146584 4986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.146677 4986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "6e5ec00cfc094015a6a59ee2238fe4a5"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "0a945928ec663271e1441a5c9022f5d0"
server_key_iv: "bfc405a3192136c5629193717e83bec1"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.146744 4986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:48.176712 4986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:48.178308 4986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:48.178462 4986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:48.178660 4986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:48.178932 4986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:48.178966 4986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.178992 4986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:48.179014 4986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.184311 4986 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:38103
I20260812 14:55:48.184350 5229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:38103 every 8 connection(s)
I20260812 14:55:48.184617 4986 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:48.186453 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 4986
I20260812 14:55:48.186554 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:55:48.186746 4204 external_mini_cluster.cc:1428] Setting key 20be7302c64c185bcb6e3076ba08dffa
Aug 12 14:55:48 dist-test-slave-6zbq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:48.193156 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.185237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:39167 (local address 127.4.27.62:43909)
0812 14:55:48.185294 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.185296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.185462 (+ 166us) server_negotiation.cc:407] Connection header received
0812 14:55:48.185896 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.185897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.185906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.185956 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:48.186871 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.187368 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.188299 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.188568 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.188638 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.190543 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:48.190545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:48.190545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:48.190553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:48.191961 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.192302 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.192303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.192303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.192369 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.192563 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.192564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.192564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.192702 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:48.192774 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.192972 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.193065 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:55:48.193174 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.185187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:43909 (local address 127.4.27.1:39167)
0812 14:55:48.185359 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.185397 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.185824 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.186012 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.186015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.186558 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:48.186786 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.186791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.187455 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.187456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.188217 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.188219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.188637 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.188638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.188742 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.189171 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:48.189181 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:48.190463 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:48.192032 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:48.192038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:48.192040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:48.192231 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:48.192430 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:48.192430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:48.192430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:48.192512 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:48.192759 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:48.192762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:48.192860 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.192930 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.193035 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260812 14:55:48.193660 5232 heartbeater.cc:347] Connected to a master server at 127.4.27.62:43909
I20260812 14:55:48.193771 5232 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:48.193930 5232 heartbeater.cc:511] Master 127.4.27.62:43909 requested a full tablet report, sending...
I20260812 14:55:48.194445 4857 ts_manager.cc:194] Registered new tserver with Master: 6e5ec00cfc094015a6a59ee2238fe4a5 (127.4.27.1:38103)
I20260812 14:55:48.195137 4857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:39167
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:55:48.209579 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:43909
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:48.287803 5240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:48.287985 5240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:48.288007 5240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:48.289054 5240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:48.289086 5240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:48.289109 5240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:48.290496 5240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:48.290673 5240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:48.290845 5240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:55:48.295497 5240 server_base.cc:1034] running on GCE node
W20260812 14:55:48.295457 5255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.295473 5252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.295470 5250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:48.295886 5240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:48.296125 5240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:48.297260 5240 hybrid_clock.cc:648] HybridClock initialized: now 1786546548297248 us; error 29 us; skew 500 ppm
Aug 12 14:55:48 dist-test-slave-6zbq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:48.299667 5240 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:55:48.300065 5240 webserver.cc:460] Webserver started at http://127.4.27.2:38321/ using document root <none> and password file <none>
I20260812 14:55:48.300256 5240 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:48.300293 5240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:48.300376 5240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:48.301293 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "68195775f4ac4789aaf01098e81ede03"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "fd075224ebdbeb5f897ee1d0ffc089d2"
server_key_iv: "d871dc20186ce1fce7482b206b759c56"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.301563 5240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "68195775f4ac4789aaf01098e81ede03"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "fd075224ebdbeb5f897ee1d0ffc089d2"
server_key_iv: "d871dc20186ce1fce7482b206b759c56"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.302706 5240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:48.303355 5280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.303499 5240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:48.303576 5240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "68195775f4ac4789aaf01098e81ede03"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "fd075224ebdbeb5f897ee1d0ffc089d2"
server_key_iv: "d871dc20186ce1fce7482b206b759c56"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.303639 5240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:48.318610 5240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:48.320541 5240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:48.320665 5240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:48.320864 5240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:48.321146 5240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:48.321177 5240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.321198 5240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:48.321221 5240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.326668 5240 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:45525
I20260812 14:55:48.326713 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:45525 every 8 connection(s)
I20260812 14:55:48.326995 5240 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:55:48 dist-test-slave-6zbq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:48.333825 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5240
I20260812 14:55:48.333947 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:55:48.334162 4204 external_mini_cluster.cc:1428] Setting key d72d780ec1f1c175a354cbfad5eaa3f8
I20260812 14:55:48.335707 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.327630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:54197 (local address 127.4.27.62:43909)
0812 14:55:48.327703 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.327705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.327829 (+ 124us) server_negotiation.cc:407] Connection header received
0812 14:55:48.328283 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.328284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.328294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.328340 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:48.329384 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.330029 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.331014 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.331199 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.331254 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.333071 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:48.333073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:48.333073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:48.333080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:48.334414 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.334806 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.334807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.334808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.334871 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.335130 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.335130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.335131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.335227 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:48.335281 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.335538 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.335622 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:55:48.335848 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.327582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:43909 (local address 127.4.27.2:54197)
0812 14:55:48.327731 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.327765 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.328208 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.328395 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.328398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.329029 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:48.329299 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.329306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.330110 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.330111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.330949 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.330952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.331263 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.331264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.331359 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.331701 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:48.331709 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:48.332982 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:48.334473 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:48.334480 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:48.334482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:48.334730 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:48.334932 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:48.334933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:48.334933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:48.335070 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:48.335289 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:48.335296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:48.335421 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.335507 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.335651 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260812 14:55:48.336740 5512 heartbeater.cc:347] Connected to a master server at 127.4.27.62:43909
I20260812 14:55:48.336829 5512 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:48.336989 5512 heartbeater.cc:511] Master 127.4.27.62:43909 requested a full tablet report, sending...
I20260812 14:55:48.337339 4857 ts_manager.cc:194] Registered new tserver with Master: 68195775f4ac4789aaf01098e81ede03 (127.4.27.2:45525)
I20260812 14:55:48.337774 4857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:54197
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:55:48.356932 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:43909
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:48.434041 5522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:48.434213 5522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:48.434235 5522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:48.435344 5522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:48.435387 5522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:48.435413 5522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:48.436822 5522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:48.437029 5522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:48.437223 5522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:48.441653 5528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.441641 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.441640 5534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:48.441805 5522 server_base.cc:1034] running on GCE node
I20260812 14:55:48.442010 5522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:48.442235 5522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:48.443372 5522 hybrid_clock.cc:648] HybridClock initialized: now 1786546548443368 us; error 39 us; skew 500 ppm
Aug 12 14:55:48 dist-test-slave-6zbq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:48.445554 5522 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:55:48.445958 5522 webserver.cc:460] Webserver started at http://127.4.27.3:41975/ using document root <none> and password file <none>
I20260812 14:55:48.446169 5522 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:48.446216 5522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:48.446334 5522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:48.447271 5522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "de883c2ec3834261abfde72148ce23d9"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "c62130ed29fa61458700699f1618da88"
server_key_iv: "c415f52e32f63ae5cfbb6dd92aa3d7e1"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.447551 5522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "de883c2ec3834261abfde72148ce23d9"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "c62130ed29fa61458700699f1618da88"
server_key_iv: "c415f52e32f63ae5cfbb6dd92aa3d7e1"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.448645 5522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:48.449280 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.449432 5522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:48.449515 5522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "de883c2ec3834261abfde72148ce23d9"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "c62130ed29fa61458700699f1618da88"
server_key_iv: "c415f52e32f63ae5cfbb6dd92aa3d7e1"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.449582 5522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:48.477115 5522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:48.479077 5522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:48.479243 5522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:48.479444 5522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:48.479708 5522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:48.479749 5522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.479779 5522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:48.479803 5522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.485090 5522 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:40405
I20260812 14:55:48.485127 5737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:40405 every 8 connection(s)
I20260812 14:55:48.485502 5522 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
Aug 12 14:55:48 dist-test-slave-6zbq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546548, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:48.491029 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5522
I20260812 14:55:48.491128 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:55:48.491278 4204 external_mini_cluster.cc:1428] Setting key ec0b1ac703d04b6fad2a43b53c32f0a2
I20260812 14:55:48.493765 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.486181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:41949 (local address 127.4.27.62:43909)
0812 14:55:48.486249 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.486251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.486494 (+ 243us) server_negotiation.cc:407] Connection header received
0812 14:55:48.486960 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.486961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.486973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.487025 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:48.487897 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.488383 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.489309 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.489500 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.489554 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.491493 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:48.491495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:48.491495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:48.491502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:48.492703 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.493034 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.493035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.493035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.493077 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.493257 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.493258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.493258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.493377 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:48.493426 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.493646 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.493704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:55:48.493899 5743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.486135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:43909 (local address 127.4.27.3:41949)
0812 14:55:48.486378 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.486420 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.486865 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.487104 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.487108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.487623 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:48.487803 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.487808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.488462 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.488463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.489224 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.489226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.489567 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.489568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.489682 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.490065 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:48.490072 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:48.491403 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:48.492771 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:48.492777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:48.492778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:48.492980 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:48.493125 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:48.493125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:48.493126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:48.493214 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:48.493426 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:48.493431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:48.493538 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.493626 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.493734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":146,"threads_started":1}
I20260812 14:55:48.494318 5739 heartbeater.cc:347] Connected to a master server at 127.4.27.62:43909
I20260812 14:55:48.494431 5739 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:48.494617 5739 heartbeater.cc:511] Master 127.4.27.62:43909 requested a full tablet report, sending...
I20260812 14:55:48.494987 4857 ts_manager.cc:194] Registered new tserver with Master: de883c2ec3834261abfde72148ce23d9 (127.4.27.3:40405)
I20260812 14:55:48.495388 4857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:41949
I20260812 14:55:48.502781 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:48.509295 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.503993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.4.27.62:43909)
0812 14:55:48.504026 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.504028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.504130 (+ 102us) server_negotiation.cc:407] Connection header received
0812 14:55:48.504191 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.504191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.504201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.504247 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:48.504783 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.505217 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.505902 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.506095 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.506155 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.507305 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:48.507306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:48.507306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:48.507312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:48.508442 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.508757 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.508758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.508758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.508801 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:48.508923 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:48.508923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:48.508923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:48.509022 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:48.509089 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.509187 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.509241 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:55:48.510838 4857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:48.512251 4857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:48.517980 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.514079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:40405 (local address 127.0.0.1:52656)
0812 14:55:48.514402 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.514414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.514466 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.514692 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.514695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.514698 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:48.514767 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.514769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.515993 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.515994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.516881 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.516883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.517697 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.517698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.517775 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.517788 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.517861 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.517910 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":67,"thread_start_us":59,"threads_started":1}
I20260812 14:55:48.517989 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.514334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.4.27.3:40405)
0812 14:55:48.514514 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.514518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.514530 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:55:48.514561 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.514564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.514579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.514656 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:48.514829 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.515884 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.516974 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.517614 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.517684 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.517735 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.517879 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.517914 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260812 14:55:48.518162 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.514053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45525 (local address 127.0.0.1:39924)
0812 14:55:48.514182 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.514219 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.514287 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.514887 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.514888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.514889 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:48.514936 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.514938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.516165 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.516166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.517254 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.517255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.517959 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.517960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.518017 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.518020 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.518049 (+ 29us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.518086 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":45,"threads_started":1}
I20260812 14:55:48.518431 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.514335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.4.27.1:38103)
0812 14:55:48.514554 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.514558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.514775 (+ 217us) server_negotiation.cc:407] Connection header received
0812 14:55:48.514889 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.514892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.514907 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.514977 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:48.515182 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.515885 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.517307 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.517987 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.518055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.518113 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.518286 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.518329 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260812 14:55:48.518643 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.514540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.4.27.2:45525)
0812 14:55:48.514718 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.514722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.514731 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:55:48.514767 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.514769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.514784 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.514888 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:48.515036 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.516060 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.517333 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.517875 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.518221 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.518490 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.518519 (+ 29us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.518560 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260812 14:55:48.518929 5626 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac33979f53c4c2c8da5a4c5273320b5 (DEFAULT_TABLE table=test-table [id=7b4970536a2d410195f7fbfa76af8685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:48.519240 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac33979f53c4c2c8da5a4c5273320b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:48.519186 5382 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac33979f53c4c2c8da5a4c5273320b5 (DEFAULT_TABLE table=test-table [id=7b4970536a2d410195f7fbfa76af8685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:48.519491 5382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac33979f53c4c2c8da5a4c5273320b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:48.521638 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.514235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:38103 (local address 127.0.0.1:38716)
0812 14:55:48.514711 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.514721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.514781 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.515062 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.515063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.515065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:48.515100 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.515101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.515975 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.515976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.517224 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.517226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.518149 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.518150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.518199 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.518203 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.518250 (+ 47us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.518291 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":38,"thread_start_us":40,"threads_started":1}
I20260812 14:55:48.521622 5764 tablet_bootstrap.cc:492] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Bootstrap starting.
I20260812 14:55:48.522109 5766 tablet_bootstrap.cc:492] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Bootstrap starting.
I20260812 14:55:48.522313 5764 tablet_bootstrap.cc:654] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:48.522248 5108 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac33979f53c4c2c8da5a4c5273320b5 (DEFAULT_TABLE table=test-table [id=7b4970536a2d410195f7fbfa76af8685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:48.522594 5108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac33979f53c4c2c8da5a4c5273320b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:48.522814 5766 tablet_bootstrap.cc:654] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:48.522986 5764 log.cc:824] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:48.523337 5766 log.cc:824] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:48.523734 5764 tablet_bootstrap.cc:492] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: No bootstrap required, opened a new log
I20260812 14:55:48.523806 5764 ts_tablet_manager.cc:1397] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:48.524034 5766 tablet_bootstrap.cc:492] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: No bootstrap required, opened a new log
I20260812 14:55:48.524156 5766 ts_tablet_manager.cc:1397] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:48.525209 5770 tablet_bootstrap.cc:492] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5: Bootstrap starting.
I20260812 14:55:48.525678 5764 raft_consensus.cc:348] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.525849 5770 tablet_bootstrap.cc:654] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:48.525858 5764 raft_consensus.cc:374] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:48.525926 5764 raft_consensus.cc:729] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de883c2ec3834261abfde72148ce23d9, State: Initialized, Role: FOLLOWER
I20260812 14:55:48.526052 5764 consensus_queue.cc:260] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.526021 5766 raft_consensus.cc:348] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.526172 5766 raft_consensus.cc:374] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:48.526240 5766 raft_consensus.cc:729] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68195775f4ac4789aaf01098e81ede03, State: Initialized, Role: FOLLOWER
I20260812 14:55:48.526356 5764 ts_tablet_manager.cc:1428] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:48.526376 5766 consensus_queue.cc:260] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.526441 5770 log.cc:824] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:48.526486 5739 heartbeater.cc:503] Master 127.4.27.62:43909 was elected leader, sending a full tablet report...
I20260812 14:55:48.526631 5512 heartbeater.cc:503] Master 127.4.27.62:43909 was elected leader, sending a full tablet report...
I20260812 14:55:48.526644 5766 ts_tablet_manager.cc:1428] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:48.527413 5770 tablet_bootstrap.cc:492] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5: No bootstrap required, opened a new log
I20260812 14:55:48.527488 5770 ts_tablet_manager.cc:1397] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:48.528805 5770 raft_consensus.cc:348] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.528928 5770 raft_consensus.cc:374] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:48.528949 5770 raft_consensus.cc:729] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5ec00cfc094015a6a59ee2238fe4a5, State: Initialized, Role: FOLLOWER
I20260812 14:55:48.529049 5770 consensus_queue.cc:260] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.529280 5770 ts_tablet_manager.cc:1428] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:48.529307 5232 heartbeater.cc:503] Master 127.4.27.62:43909 was elected leader, sending a full tablet report...
I20260812 14:55:48.558488 5778 raft_consensus.cc:482] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:48.558568 5778 raft_consensus.cc:504] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.559103 5778 leader_election.cc:288] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68195775f4ac4789aaf01098e81ede03 (127.4.27.2:45525), de883c2ec3834261abfde72148ce23d9 (127.4.27.3:40405)
I20260812 14:55:48.561509 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.559064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:33977 (local address 127.4.27.2:45525)
0812 14:55:48.559103 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.559105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.559113 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:55:48.559136 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.559137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.559148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.559203 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:48.559375 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.559892 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.560772 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.561186 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.561240 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.561249 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.561412 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.561447 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:55:48.561513 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.559153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:47403 (local address 127.4.27.3:40405)
0812 14:55:48.559191 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.559193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.559226 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:55:48.559328 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.559330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.559341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.559392 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:48.559527 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.559965 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.560856 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.561216 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.561281 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.561291 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.561422 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.561456 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:55:48.561532 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.558963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:45525 (local address 127.4.27.1:33977)
0812 14:55:48.559032 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.559043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.559080 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.559236 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.559237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.559239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:48.559302 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.559303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.559967 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.559968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.560686 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.560688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.561273 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.561274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.561345 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.561348 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.561390 (+ 42us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.561429 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 14:55:48.561547 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.558963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:40405 (local address 127.4.27.1:47403)
0812 14:55:48.559166 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.559176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:48.559251 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:48.559400 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:48.559400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:48.559414 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:48.559470 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.559472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.560035 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.560036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.560780 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:48.560782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.561297 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.561298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:48.561356 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.561360 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:48.561410 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:55:48.561447 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":10,"thread_start_us":101,"threads_started":1}
I20260812 14:55:48.561851 5426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac33979f53c4c2c8da5a4c5273320b5" candidate_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68195775f4ac4789aaf01098e81ede03" is_pre_election: true
I20260812 14:55:48.561843 5664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac33979f53c4c2c8da5a4c5273320b5" candidate_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de883c2ec3834261abfde72148ce23d9" is_pre_election: true
I20260812 14:55:48.561972 5426 raft_consensus.cc:2454] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ec00cfc094015a6a59ee2238fe4a5 in term 0.
I20260812 14:55:48.561972 5664 raft_consensus.cc:2454] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5ec00cfc094015a6a59ee2238fe4a5 in term 0.
I20260812 14:55:48.562204 5011 leader_election.cc:302] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5ec00cfc094015a6a59ee2238fe4a5, de883c2ec3834261abfde72148ce23d9; no voters:
I20260812 14:55:48.562327 5778 raft_consensus.cc:2781] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:48.562418 5778 raft_consensus.cc:482] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:48.562438 5778 raft_consensus.cc:3037] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:48.563052 5778 raft_consensus.cc:504] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.563184 5778 leader_election.cc:288] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [CANDIDATE]: Term 1 election: Requested vote from peers 68195775f4ac4789aaf01098e81ede03 (127.4.27.2:45525), de883c2ec3834261abfde72148ce23d9 (127.4.27.3:40405)
I20260812 14:55:48.563328 5664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac33979f53c4c2c8da5a4c5273320b5" candidate_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de883c2ec3834261abfde72148ce23d9"
I20260812 14:55:48.563342 5426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac33979f53c4c2c8da5a4c5273320b5" candidate_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68195775f4ac4789aaf01098e81ede03"
I20260812 14:55:48.563400 5664 raft_consensus.cc:3037] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:48.563403 5426 raft_consensus.cc:3037] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:48.564054 5426 raft_consensus.cc:2454] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ec00cfc094015a6a59ee2238fe4a5 in term 1.
I20260812 14:55:48.564090 5664 raft_consensus.cc:2454] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5ec00cfc094015a6a59ee2238fe4a5 in term 1.
I20260812 14:55:48.564247 5014 leader_election.cc:302] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68195775f4ac4789aaf01098e81ede03, 6e5ec00cfc094015a6a59ee2238fe4a5; no voters:
I20260812 14:55:48.564339 5778 raft_consensus.cc:2781] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:48.564467 5778 raft_consensus.cc:686] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5ec00cfc094015a6a59ee2238fe4a5, State: Running, Role: LEADER
I20260812 14:55:48.564551 5778 consensus_queue.cc:237] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } }
I20260812 14:55:48.565258 4857 catalog_manager.cc:5495] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e5ec00cfc094015a6a59ee2238fe4a5 (127.4.27.1). New cstate: current_term: 1 leader_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5ec00cfc094015a6a59ee2238fe4a5" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 38103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 } health_report { overall_health: UNKNOWN } } }
I20260812 14:55:48.574791 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.572626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.4.27.1:38103)
0812 14:55:48.572671 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.572673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.572681 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:55:48.572739 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.572740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.572756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.572816 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:48.572930 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.573351 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.574045 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.574240 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.574292 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.574481 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:48.574565 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:48.574649 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.574700 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.574708 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:55:48.576224 5426 raft_consensus.cc:1261] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 1 FOLLOWER]: Refusing update from remote peer 6e5ec00cfc094015a6a59ee2238fe4a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:48.576234 5664 raft_consensus.cc:1261] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 1 FOLLOWER]: Refusing update from remote peer 6e5ec00cfc094015a6a59ee2238fe4a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:48.576524 5780 consensus_queue.cc:1035] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de883c2ec3834261abfde72148ce23d9" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:48.576639 5778 consensus_queue.cc:1035] T 2ac33979f53c4c2c8da5a4c5273320b5 P 6e5ec00cfc094015a6a59ee2238fe4a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68195775f4ac4789aaf01098e81ede03" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 14:55:48.577481 5514 tablet.cc:2369] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:48.579458 5786 mvcc.cc:205] Tried to move back new op lower bound from 7317694662966042624 to 7317694662920900608. Current Snapshot: MvccSnapshot[applied={T|T < 7317694662966042624 or (T in {7317694662966042624})}]
I20260812 14:55:48.579620 5791 mvcc.cc:205] Tried to move back new op lower bound from 7317694662966042624 to 7317694662920900608. Current Snapshot: MvccSnapshot[applied={T|T < 7317694662966042624}]
I20260812 14:55:48.580740 4856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:48.580829 4856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:48.581522 4856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10af38ed5abc4ae3a5148b2e4802267c: Sending DeleteTablet for 3 replicas of tablet 2ac33979f53c4c2c8da5a4c5273320b5
I20260812 14:55:48.581872 5626 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac33979f53c4c2c8da5a4c5273320b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:48 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:52656
I20260812 14:55:48.581872 5382 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac33979f53c4c2c8da5a4c5273320b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:48 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:39924
I20260812 14:55:48.582080 5805 tablet_replica.cc:330] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: stopping tablet replica
I20260812 14:55:48.582079 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4986
I20260812 14:55:48.582070 5108 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac33979f53c4c2c8da5a4c5273320b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:48 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:38716
I20260812 14:55:48.582160 5805 raft_consensus.cc:2229] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:48.594270 5806 tablet_replica.cc:330] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: stopping tablet replica
I20260812 14:55:48.594439 5806 raft_consensus.cc:2229] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:48.594547 5806 raft_consensus.cc:2258] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:48.594939 5806 ts_tablet_manager.cc:1905] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:48.596299 5806 ts_tablet_manager.cc:1918] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:48.596360 5806 log.cc:1196] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/2ac33979f53c4c2c8da5a4c5273320b5
I20260812 14:55:48.596582 5806 ts_tablet_manager.cc:1939] T 2ac33979f53c4c2c8da5a4c5273320b5 P 68195775f4ac4789aaf01098e81ede03: Deleting consensus metadata
I20260812 14:55:48.596957 4829 catalog_manager.cc:4841] TS 68195775f4ac4789aaf01098e81ede03 (127.4.27.2:45525): tablet 2ac33979f53c4c2c8da5a4c5273320b5 (table test-table [id=7b4970536a2d410195f7fbfa76af8685]) successfully deleted
I20260812 14:55:48.598464 5805 raft_consensus.cc:2258] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:48.598817 5805 ts_tablet_manager.cc:1905] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 14:55:48.599869 4829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:38103 (error 108)
I20260812 14:55:48.599938 5805 ts_tablet_manager.cc:1918] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:48.600008 5805 log.cc:1196] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/2ac33979f53c4c2c8da5a4c5273320b5
I20260812 14:55:48.600162 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5240
I20260812 14:55:48.601927 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.600311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:38103 (local address 127.0.0.1:38728)
0812 14:55:48.601853 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:48.601896 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:38103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1519}
W20260812 14:55:48.603026 4829 catalog_manager.cc:4584] TS 6e5ec00cfc094015a6a59ee2238fe4a5 (127.4.27.1:38103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ac33979f53c4c2c8da5a4c5273320b5: Network error: Client connection negotiation failed: client connection to 127.4.27.1:38103: connect: Connection refused (error 111)
I20260812 14:55:48.612252 5805 ts_tablet_manager.cc:1939] T 2ac33979f53c4c2c8da5a4c5273320b5 P de883c2ec3834261abfde72148ce23d9: Deleting consensus metadata
I20260812 14:55:48.612769 4827 catalog_manager.cc:4841] TS de883c2ec3834261abfde72148ce23d9 (127.4.27.3:40405): tablet 2ac33979f53c4c2c8da5a4c5273320b5 (table test-table [id=7b4970536a2d410195f7fbfa76af8685]) successfully deleted
I20260812 14:55:48.618391 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5522
I20260812 14:55:48.631268 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 4803
2026-08-12T14:55:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T14:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:48Z Disabled control of system clock
I20260812 14:55:48.663159 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46859
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:48.742020 5816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:48.742228 5816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:48.742256 5816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:48.743407 5816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:48.743449 5816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:48.743465 5816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:48.743481 5816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:48.743497 5816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:48.744803 5816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:48.745033 5816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:48.745280 5816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:48.749712 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.749729 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.749890 5834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:48.749933 5816 server_base.cc:1034] running on GCE node
I20260812 14:55:48.750166 5816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:48.750494 5816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:48.751628 5816 hybrid_clock.cc:648] HybridClock initialized: now 1786546548751616 us; error 30 us; skew 500 ppm
I20260812 14:55:48.752715 5816 webserver.cc:460] Webserver started at http://127.4.27.62:44213/ using document root <none> and password file <none>
I20260812 14:55:48.752926 5816 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:48.752971 5816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:48.753096 5816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:48.754055 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "62c8e6682f95424bbeab0e7e770b7106"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "e964a358717de2ebbfd58cae973d9883"
server_key_iv: "383e077e61d21d40b228224a0d25346b"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.754422 5816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "62c8e6682f95424bbeab0e7e770b7106"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "e964a358717de2ebbfd58cae973d9883"
server_key_iv: "383e077e61d21d40b228224a0d25346b"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.755561 5816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:48.756214 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.756388 5816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:48.756474 5816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "62c8e6682f95424bbeab0e7e770b7106"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "e964a358717de2ebbfd58cae973d9883"
server_key_iv: "383e077e61d21d40b228224a0d25346b"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.756531 5816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:48.788965 5816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:48.789229 5816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:48.789326 5816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:48.792920 5816 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46859
I20260812 14:55:48.792955 5934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46859 every 8 connection(s)
I20260812 14:55:48.793277 5816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:48.794211 5935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:48.796473 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106: Bootstrap starting.
I20260812 14:55:48.797034 5935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:48.797116 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5816
I20260812 14:55:48.797227 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:48.797339 5935 log.cc:824] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:48.797405 4204 external_mini_cluster.cc:1428] Setting key c34e89725b57c8c195ffa684bd17b2a9
I20260812 14:55:48.798051 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106: No bootstrap required, opened a new log
I20260812 14:55:48.800060 5935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c8e6682f95424bbeab0e7e770b7106" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46859 } }
I20260812 14:55:48.800195 5935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:48.800227 5935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c8e6682f95424bbeab0e7e770b7106, State: Initialized, Role: FOLLOWER
I20260812 14:55:48.800349 5935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c8e6682f95424bbeab0e7e770b7106" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46859 } }
I20260812 14:55:48.800420 5935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:48.800450 5935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:48.800498 5935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:48.801009 5935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c8e6682f95424bbeab0e7e770b7106" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46859 } }
I20260812 14:55:48.801119 5935 leader_election.cc:302] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62c8e6682f95424bbeab0e7e770b7106; no voters:
I20260812 14:55:48.801287 5935 leader_election.cc:288] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:48.801321 5943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:48.801434 5943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [term 1 LEADER]: Becoming Leader. State: Replica: 62c8e6682f95424bbeab0e7e770b7106, State: Running, Role: LEADER
I20260812 14:55:48.801517 5943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c8e6682f95424bbeab0e7e770b7106" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46859 } }
I20260812 14:55:48.801853 5935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:48.801889 5944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62c8e6682f95424bbeab0e7e770b7106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c8e6682f95424bbeab0e7e770b7106" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46859 } } }
I20260812 14:55:48.801985 5944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:48.801903 5946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62c8e6682f95424bbeab0e7e770b7106. Latest consensus state: current_term: 1 leader_uuid: "62c8e6682f95424bbeab0e7e770b7106" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c8e6682f95424bbeab0e7e770b7106" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46859 } } }
I20260812 14:55:48.802202 5946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:48.802219 5953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:48.802667 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:48.797834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.4.27.62:46859)
0812 14:55:48.798038 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:48.798044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:48.798067 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:55:48.798533 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:48.798540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:48.798595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:48.798722 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:48.798925 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.799793 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.800644 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:48.801041 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:48.801160 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:48.801822 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:48.801826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:48.801826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:48.802020 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:48.802063 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:48.802079 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:48.802385 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:48.802417 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 14:55:48.802494 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260812 14:55:48.802721 5953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:48.804318 5953 catalog_manager.cc:1347] Generated new cluster ID: 4027b5476e9a4d0980d9647469872080
I20260812 14:55:48.804375 5953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:48.826678 5953 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:48.827198 5953 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:48.838310 5953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106: Generated new TSK 0
I20260812 14:55:48.838496 5953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:48.844413 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46859
--builtin_ntp_servers=127.4.27.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:48.949249 5977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:48.949440 5977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:48.949460 5977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:48.950627 5977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:48.950670 5977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:48.950695 5977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:48.952020 5977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:48.952246 5977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:48.952512 5977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:48.957484 5990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.957507 5985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:48.957517 5988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:48.957623 5977 server_base.cc:1034] running on GCE node
I20260812 14:55:48.958032 5977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:48.958225 5977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:48.959357 5977 hybrid_clock.cc:648] HybridClock initialized: now 1786546548959343 us; error 30 us; skew 500 ppm
I20260812 14:55:48.960491 5977 webserver.cc:460] Webserver started at http://127.4.27.1:40865/ using document root <none> and password file <none>
I20260812 14:55:48.960696 5977 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:48.960752 5977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:48.960871 5977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:48.961871 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "d2a697b1d73c49c581556f06c08ff9f9"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "d1bbc3af4520cd8d5e6a486ccfd6c61a"
server_key_iv: "5122d3df8c741dfa9127daf618cdce99"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.962186 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "d2a697b1d73c49c581556f06c08ff9f9"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "d1bbc3af4520cd8d5e6a486ccfd6c61a"
server_key_iv: "5122d3df8c741dfa9127daf618cdce99"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.963413 5977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:48.964089 6000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.964253 5977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:48.964334 5977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "d2a697b1d73c49c581556f06c08ff9f9"
format_stamp: "Formatted at 2026-08-12 14:55:48 on dist-test-slave-6zbq"
server_key: "d1bbc3af4520cd8d5e6a486ccfd6c61a"
server_key_iv: "5122d3df8c741dfa9127daf618cdce99"
server_key_version: "encryptionkey@0"
I20260812 14:55:48.964401 5977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:48.993620 5977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:48.993880 5977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:48.993976 5977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:48.994149 5977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:48.994457 5977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:48.994493 5977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.994519 5977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:48.994534 5977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:48.999935 5977 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:45727
I20260812 14:55:49.000005 6196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:45727 every 8 connection(s)
I20260812 14:55:49.000303 5977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:49.000885 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 5977
I20260812 14:55:49.000982 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:55:49.001237 4204 external_mini_cluster.cc:1428] Setting key fb91e9856f0ae7a774406246e5fcec30
I20260812 14:55:49.002202 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46859
--builtin_ntp_servers=127.4.27.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:55:49.005967 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.001447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:57395 (local address 127.4.27.62:46859)
0812 14:55:49.001506 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.001508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.001518 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:55:49.001980 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.001981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.001989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.002024 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:49.002993 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.003474 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.004593 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.004798 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.004857 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.005737 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:49.005738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:49.005738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:49.005743 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:49.005758 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:49.005762 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:49.005807 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.005886 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.005907 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 14:55:49.006283 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.000989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46859 (local address 127.4.27.1:57395)
0812 14:55:49.001229 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.001324 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.001882 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.002105 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.002109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.002120 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:49.002907 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.002915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.003570 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.003571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.004502 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.004505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.004877 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.004878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.005027 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.005566 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:49.005567 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:49.005605 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:49.005633 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:49.005818 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:49.005823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:49.005826 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.005885 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.005989 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260812 14:55:49.006815 6197 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46859
I20260812 14:55:49.006908 6197 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:49.007081 6197 heartbeater.cc:511] Master 127.4.27.62:46859 requested a full tablet report, sending...
I20260812 14:55:49.007557 5869 ts_manager.cc:194] Registered new tserver with Master: d2a697b1d73c49c581556f06c08ff9f9 (127.4.27.1:45727)
I20260812 14:55:49.008157 5869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.1:57395
W20260812 14:55:49.084921 6205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:49.085145 6205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:49.085179 6205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:49.086392 6205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:49.086431 6205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:49.086465 6205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:49.087827 6205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:49.088066 6205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:49.088347 6205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:49.093325 6216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:49.093461 6219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:49.093392 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:49.093536 6205 server_base.cc:1034] running on GCE node
I20260812 14:55:49.093789 6205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:49.094022 6205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:49.095268 6205 hybrid_clock.cc:648] HybridClock initialized: now 1786546549095276 us; error 70 us; skew 500 ppm
I20260812 14:55:49.096511 6205 webserver.cc:460] Webserver started at http://127.4.27.2:39685/ using document root <none> and password file <none>
I20260812 14:55:49.096721 6205 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:49.096764 6205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:49.096903 6205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:49.097944 6205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "e404afe42de54a309a6db10e16aa2eef"
format_stamp: "Formatted at 2026-08-12 14:55:49 on dist-test-slave-6zbq"
server_key: "204679c0bcc741f626a27d6aaecd4b07"
server_key_iv: "83616e9a75603792c6481603ca2e2d90"
server_key_version: "encryptionkey@0"
I20260812 14:55:49.098260 6205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "e404afe42de54a309a6db10e16aa2eef"
format_stamp: "Formatted at 2026-08-12 14:55:49 on dist-test-slave-6zbq"
server_key: "204679c0bcc741f626a27d6aaecd4b07"
server_key_iv: "83616e9a75603792c6481603ca2e2d90"
server_key_version: "encryptionkey@0"
I20260812 14:55:49.099517 6205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:49.100198 6232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:49.100340 6205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:49.100422 6205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "e404afe42de54a309a6db10e16aa2eef"
format_stamp: "Formatted at 2026-08-12 14:55:49 on dist-test-slave-6zbq"
server_key: "204679c0bcc741f626a27d6aaecd4b07"
server_key_iv: "83616e9a75603792c6481603ca2e2d90"
server_key_version: "encryptionkey@0"
I20260812 14:55:49.100483 6205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:49.120487 6205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:49.120779 6205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:49.120905 6205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:49.121135 6205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:49.121448 6205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:49.121483 6205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:49.121512 6205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:49.121531 6205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:49.126858 6205 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:36667
I20260812 14:55:49.126919 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:36667 every 8 connection(s)
I20260812 14:55:49.127211 6205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:55:49.128432 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6205
I20260812 14:55:49.128531 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:55:49.128715 4204 external_mini_cluster.cc:1428] Setting key 0a6c53ea96ed6bdc0c88574084e7612d
I20260812 14:55:49.129635 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46859
--builtin_ntp_servers=127.4.27.20:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 14:55:49.132356 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.127879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:45267 (local address 127.4.27.62:46859)
0812 14:55:49.127938 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.127940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.128125 (+ 185us) server_negotiation.cc:407] Connection header received
0812 14:55:49.128664 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.128666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.128678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.128733 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:49.129542 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.130038 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.131026 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.131233 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.131289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.132059 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:49.132060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:49.132061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:49.132066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:49.132082 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:49.132087 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:49.132135 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.132260 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.132284 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:55:49.132515 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.127836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46859 (local address 127.4.27.2:45267)
0812 14:55:49.128020 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.128051 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.128553 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.128802 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.128805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.128813 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:49.129456 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.129463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.130123 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.130124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.130942 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.130944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.131306 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.131307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.131443 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.131926 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:49.131927 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:49.131954 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:49.131976 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:49.132164 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:49.132171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:49.132175 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.132247 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.132330 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260812 14:55:49.132985 6435 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46859
I20260812 14:55:49.133080 6435 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:49.133265 6435 heartbeater.cc:511] Master 127.4.27.62:46859 requested a full tablet report, sending...
I20260812 14:55:49.133616 5869 ts_manager.cc:194] Registered new tserver with Master: e404afe42de54a309a6db10e16aa2eef (127.4.27.2:36667)
I20260812 14:55:49.134023 5869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.2:45267
W20260812 14:55:49.210284 6439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:49.210507 6439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:49.210527 6439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:49.211608 6439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:49.211654 6439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:49.211678 6439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:49.212960 6439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:49.213157 6439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:49.213400 6439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:49.217967 6456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:49.218005 6452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:49.217942 6448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:49.218151 6439 server_base.cc:1034] running on GCE node
I20260812 14:55:49.218364 6439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:49.218570 6439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:49.219713 6439 hybrid_clock.cc:648] HybridClock initialized: now 1786546549219702 us; error 26 us; skew 500 ppm
I20260812 14:55:49.220816 6439 webserver.cc:460] Webserver started at http://127.4.27.3:34889/ using document root <none> and password file <none>
I20260812 14:55:49.220988 6439 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:49.221025 6439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:49.221125 6439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:49.222065 6439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "0349a0dd2d0f4e14a5791eafef5aa023"
format_stamp: "Formatted at 2026-08-12 14:55:49 on dist-test-slave-6zbq"
server_key: "a79e542f154fca122013635a2b4c5601"
server_key_iv: "6487ba0922bc7f4fe2339d7a3e7b01c3"
server_key_version: "encryptionkey@0"
I20260812 14:55:49.222388 6439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "0349a0dd2d0f4e14a5791eafef5aa023"
format_stamp: "Formatted at 2026-08-12 14:55:49 on dist-test-slave-6zbq"
server_key: "a79e542f154fca122013635a2b4c5601"
server_key_iv: "6487ba0922bc7f4fe2339d7a3e7b01c3"
server_key_version: "encryptionkey@0"
I20260812 14:55:49.223489 6439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:49.224067 6465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:49.224200 6439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:49.224282 6439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "0349a0dd2d0f4e14a5791eafef5aa023"
format_stamp: "Formatted at 2026-08-12 14:55:49 on dist-test-slave-6zbq"
server_key: "a79e542f154fca122013635a2b4c5601"
server_key_iv: "6487ba0922bc7f4fe2339d7a3e7b01c3"
server_key_version: "encryptionkey@0"
I20260812 14:55:49.224350 6439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:49.254232 6439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:49.254614 6439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:49.254753 6439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:49.254983 6439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:49.255347 6439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:49.255383 6439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:49.255414 6439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:49.255437 6439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:49.260775 6439 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:38509
I20260812 14:55:49.260838 6672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:38509 every 8 connection(s)
I20260812 14:55:49.261157 6439 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:55:49.265479 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6439
I20260812 14:55:49.265594 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:55:49.265784 4204 external_mini_cluster.cc:1428] Setting key 8db47e053f65e0380a39497001667c2b
I20260812 14:55:49.266384 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.261827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:57295 (local address 127.4.27.62:46859)
0812 14:55:49.261888 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.261891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.262126 (+ 235us) server_negotiation.cc:407] Connection header received
0812 14:55:49.262785 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.262786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.262800 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.262853 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:49.263532 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.264031 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.264965 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.265157 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.265214 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.266075 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:49.266077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:49.266078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:49.266082 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:49.266102 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:49.266107 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:49.266162 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.266254 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.266280 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:55:49.266636 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.261822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46859 (local address 127.4.27.3:57295)
0812 14:55:49.262010 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.262045 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.262674 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.262924 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.262928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.262936 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:49.263440 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.263445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.264116 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.264116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.264883 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.264885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.265227 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.265228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.265400 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.265936 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:49.265936 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:49.265965 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:49.265987 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:49.266171 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:49.266176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:49.266179 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.266250 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.266375 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260812 14:55:49.267076 6673 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46859
I20260812 14:55:49.267179 6673 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:49.267350 6673 heartbeater.cc:511] Master 127.4.27.62:46859 requested a full tablet report, sending...
I20260812 14:55:49.267673 5869 ts_manager.cc:194] Registered new tserver with Master: 0349a0dd2d0f4e14a5791eafef5aa023 (127.4.27.3:38509)
I20260812 14:55:49.268021 5869 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.3:57295
I20260812 14:55:49.277546 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:49.281963 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.279221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.4.27.62:46859)
0812 14:55:49.279291 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.279292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.279358 (+ 66us) server_negotiation.cc:407] Connection header received
0812 14:55:49.279422 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.279422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.279430 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.279470 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:49.279677 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.280127 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.280890 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.281072 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.281140 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.281727 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:49.281728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:49.281729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:49.281733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:49.281747 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:49.281751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:49.281828 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.281876 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.281885 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 14:55:49.283401 5869 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:49.284885 5869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:49.290735 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.286892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.4.27.3:38509)
0812 14:55:49.287146 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.287150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.287307 (+ 157us) server_negotiation.cc:407] Connection header received
0812 14:55:49.287378 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.287380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.287395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.287466 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:49.287707 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.288397 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.289701 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.290327 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.290397 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.290492 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.290623 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.290658 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260812 14:55:49.290719 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.286795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38509 (local address 127.0.0.1:32868)
0812 14:55:49.287243 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.287254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.287314 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.287524 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.287524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.287526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:49.287626 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.287628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.288472 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.288472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.289599 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.289602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.290435 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.290436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.290514 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.290526 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.290587 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.290644 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":43,"thread_start_us":68,"threads_started":1}
I20260812 14:55:49.290969 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.286794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:45727 (local address 127.0.0.1:52232)
0812 14:55:49.286895 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.286929 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.286988 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.287419 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.287422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.287424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:49.287491 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.287493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.288777 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.288777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.289854 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.289857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.290775 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.290776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.290838 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.290841 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.290871 (+ 30us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.290904 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260812 14:55:49.291385 6552 tablet_service.cc:1467] Processing CreateTablet for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a (DEFAULT_TABLE table=test-table [id=ac6df5e0f5494a019d46b8f44636bd27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:49.291647 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.286997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:36667 (local address 127.0.0.1:40574)
0812 14:55:49.287268 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.287276 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.287327 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.287497 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.287498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.287500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:49.287601 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.287603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.288825 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.288826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.289583 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.289586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.290563 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.290564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.290625 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.290632 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.290683 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.290725 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":64,"threads_started":1}
I20260812 14:55:49.291728 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.287074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.4.27.1:45727)
0812 14:55:49.287240 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.287244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.287256 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:55:49.287295 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.287297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.287313 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.287413 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:49.287571 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.288688 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.289935 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.290687 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.291079 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.291389 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.291446 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.291519 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260812 14:55:49.291724 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:49.291967 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.287097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.4.27.2:36667)
0812 14:55:49.287241 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.287245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.287377 (+ 132us) server_negotiation.cc:407] Connection header received
0812 14:55:49.287418 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.287420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.287432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.287500 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:49.287685 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.288746 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.289668 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.290371 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.290436 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.290524 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.291840 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.291890 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260812 14:55:49.292757 6078 tablet_service.cc:1467] Processing CreateTablet for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a (DEFAULT_TABLE table=test-table [id=ac6df5e0f5494a019d46b8f44636bd27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:49.292834 6306 tablet_service.cc:1467] Processing CreateTablet for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a (DEFAULT_TABLE table=test-table [id=ac6df5e0f5494a019d46b8f44636bd27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:49.293138 6078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:49.293296 6306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:49.295010 6701 tablet_bootstrap.cc:492] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Bootstrap starting.
I20260812 14:55:49.295658 6701 tablet_bootstrap.cc:654] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:49.295954 6701 log.cc:824] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:49.296012 6705 tablet_bootstrap.cc:492] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9: Bootstrap starting.
I20260812 14:55:49.296604 6705 tablet_bootstrap.cc:654] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:49.296805 6701 tablet_bootstrap.cc:492] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: No bootstrap required, opened a new log
I20260812 14:55:49.296860 6701 ts_tablet_manager.cc:1397] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:49.296913 6705 log.cc:824] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:49.297224 6704 tablet_bootstrap.cc:492] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Bootstrap starting.
I20260812 14:55:49.297605 6705 tablet_bootstrap.cc:492] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9: No bootstrap required, opened a new log
I20260812 14:55:49.297675 6705 ts_tablet_manager.cc:1397] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:49.297860 6704 tablet_bootstrap.cc:654] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:49.298158 6704 log.cc:824] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:49.298942 6704 tablet_bootstrap.cc:492] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: No bootstrap required, opened a new log
I20260812 14:55:49.299017 6704 ts_tablet_manager.cc:1397] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:55:49.299024 6701 raft_consensus.cc:348] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.299189 6701 raft_consensus.cc:374] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:49.299230 6701 raft_consensus.cc:729] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0349a0dd2d0f4e14a5791eafef5aa023, State: Initialized, Role: FOLLOWER
I20260812 14:55:49.299342 6701 consensus_queue.cc:260] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.299636 6701 ts_tablet_manager.cc:1428] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:55:49.299916 6705 raft_consensus.cc:348] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.300037 6705 raft_consensus.cc:374] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:49.300068 6705 raft_consensus.cc:729] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a697b1d73c49c581556f06c08ff9f9, State: Initialized, Role: FOLLOWER
I20260812 14:55:49.300258 6705 consensus_queue.cc:260] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.300534 6705 ts_tablet_manager.cc:1428] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:55:49.300590 6197 heartbeater.cc:503] Master 127.4.27.62:46859 was elected leader, sending a full tablet report...
I20260812 14:55:49.301156 6704 raft_consensus.cc:348] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.301306 6704 raft_consensus.cc:374] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:49.301340 6704 raft_consensus.cc:729] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e404afe42de54a309a6db10e16aa2eef, State: Initialized, Role: FOLLOWER
I20260812 14:55:49.301445 6704 consensus_queue.cc:260] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.301673 6704 ts_tablet_manager.cc:1428] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260812 14:55:49.301716 6435 heartbeater.cc:503] Master 127.4.27.62:46859 was elected leader, sending a full tablet report...
I20260812 14:55:49.301826 6673 heartbeater.cc:503] Master 127.4.27.62:46859 was elected leader, sending a full tablet report...
I20260812 14:55:49.367985 6716 raft_consensus.cc:482] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:49.368104 6716 raft_consensus.cc:504] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.368436 6716 leader_election.cc:288] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0349a0dd2d0f4e14a5791eafef5aa023 (127.4.27.3:38509), d2a697b1d73c49c581556f06c08ff9f9 (127.4.27.1:45727)
I20260812 14:55:49.370977 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.368648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:49929 (local address 127.4.27.3:38509)
0812 14:55:49.368692 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.368694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.368703 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:55:49.368754 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.368755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.368763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.368805 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:49.368946 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.369413 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.370296 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.370675 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.370737 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.370746 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.370884 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.370916 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:55:49.370978 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.368591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:38509 (local address 127.4.27.2:49929)
0812 14:55:49.368650 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.368660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.368702 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.368818 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.368818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.368820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:49.368888 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.368890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.369480 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.369481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.370203 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.370204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.370745 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.370746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.370823 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.370826 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.370871 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.370911 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260812 14:55:49.371276 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.368690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:33275 (local address 127.4.27.1:45727)
0812 14:55:49.368736 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.368738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.368865 (+ 127us) server_negotiation.cc:407] Connection header received
0812 14:55:49.368913 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.368914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.368925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.368972 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:49.369102 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.369578 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.370489 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.370848 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.370907 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.370916 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.371174 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.371211 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:55:49.371299 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7166cf9eea94c0aa5dac6a6ccb68a6a" candidate_uuid: "e404afe42de54a309a6db10e16aa2eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" is_pre_election: true
I20260812 14:55:49.371330 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.368627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:45727 (local address 127.4.27.2:33275)
0812 14:55:49.368794 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.368803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:49.368857 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:49.368979 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:49.368980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:49.368981 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:49.369043 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.369045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.369652 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.369653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.370415 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:49.370417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.370928 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.370929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:49.370995 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.370998 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:49.371048 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 14:55:49.371258 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
I20260812 14:55:49.371438 6591 raft_consensus.cc:2454] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e404afe42de54a309a6db10e16aa2eef in term 0.
I20260812 14:55:49.371675 6236 leader_election.cc:302] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0349a0dd2d0f4e14a5791eafef5aa023, e404afe42de54a309a6db10e16aa2eef; no voters:
I20260812 14:55:49.371670 6113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7166cf9eea94c0aa5dac6a6ccb68a6a" candidate_uuid: "e404afe42de54a309a6db10e16aa2eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a697b1d73c49c581556f06c08ff9f9" is_pre_election: true
I20260812 14:55:49.371795 6113 raft_consensus.cc:2454] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e404afe42de54a309a6db10e16aa2eef in term 0.
I20260812 14:55:49.371814 6716 raft_consensus.cc:2781] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:49.371881 6716 raft_consensus.cc:482] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:49.371906 6716 raft_consensus.cc:3037] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:49.372588 6716 raft_consensus.cc:504] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.372706 6716 leader_election.cc:288] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [CANDIDATE]: Term 1 election: Requested vote from peers 0349a0dd2d0f4e14a5791eafef5aa023 (127.4.27.3:38509), d2a697b1d73c49c581556f06c08ff9f9 (127.4.27.1:45727)
I20260812 14:55:49.372850 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7166cf9eea94c0aa5dac6a6ccb68a6a" candidate_uuid: "e404afe42de54a309a6db10e16aa2eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0349a0dd2d0f4e14a5791eafef5aa023"
I20260812 14:55:49.372889 6113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7166cf9eea94c0aa5dac6a6ccb68a6a" candidate_uuid: "e404afe42de54a309a6db10e16aa2eef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a697b1d73c49c581556f06c08ff9f9"
I20260812 14:55:49.372922 6591 raft_consensus.cc:3037] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:49.372946 6113 raft_consensus.cc:3037] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:49.373587 6591 raft_consensus.cc:2454] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e404afe42de54a309a6db10e16aa2eef in term 1.
I20260812 14:55:49.373654 6113 raft_consensus.cc:2454] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e404afe42de54a309a6db10e16aa2eef in term 1.
I20260812 14:55:49.373764 6236 leader_election.cc:302] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0349a0dd2d0f4e14a5791eafef5aa023, e404afe42de54a309a6db10e16aa2eef; no voters:
I20260812 14:55:49.373852 6716 raft_consensus.cc:2781] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:49.373999 6716 raft_consensus.cc:686] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 1 LEADER]: Becoming Leader. State: Replica: e404afe42de54a309a6db10e16aa2eef, State: Running, Role: LEADER
I20260812 14:55:49.374081 6716 consensus_queue.cc:237] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } }
I20260812 14:55:49.374862 5868 catalog_manager.cc:5495] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef reported cstate change: term changed from 0 to 1, leader changed from <none> to e404afe42de54a309a6db10e16aa2eef (127.4.27.2). New cstate: current_term: 1 leader_uuid: "e404afe42de54a309a6db10e16aa2eef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e404afe42de54a309a6db10e16aa2eef" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 36667 } health_report { overall_health: HEALTHY } } }
W20260812 14:55:49.377693 6437 tablet.cc:2369] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:49.385250 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.383336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.4.27.2:36667)
0812 14:55:49.383377 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:49.383379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:49.383386 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:55:49.383416 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:49.383417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:49.383433 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:49.383487 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:49.383585 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.384008 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.384680 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:49.384847 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:49.384890 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:49.384996 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:49.385072 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:49.385134 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:49.385175 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 14:55:49.385189 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:55:49.386816 6591 raft_consensus.cc:1261] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 1 FOLLOWER]: Refusing update from remote peer e404afe42de54a309a6db10e16aa2eef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:49.386816 6113 raft_consensus.cc:1261] T f7166cf9eea94c0aa5dac6a6ccb68a6a P d2a697b1d73c49c581556f06c08ff9f9 [term 1 FOLLOWER]: Refusing update from remote peer e404afe42de54a309a6db10e16aa2eef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:49.387157 6719 consensus_queue.cc:1035] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "0349a0dd2d0f4e14a5791eafef5aa023" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 38509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:49.387287 6716 consensus_queue.cc:1035] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2a697b1d73c49c581556f06c08ff9f9" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:49.389230 6723 mvcc.cc:205] Tried to move back new op lower bound from 7317694666286002176 to 7317694666236928000. Current Snapshot: MvccSnapshot[applied={T|T < 7317694666286002176}]
I20260812 14:55:49.390414 6724 mvcc.cc:205] Tried to move back new op lower bound from 7317694666286002176 to 7317694666236928000. Current Snapshot: MvccSnapshot[applied={T|T < 7317694666286002176 or (T in {7317694666286002176})}]
I20260812 14:55:49.391747 5868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:49.391839 5868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:49.392501 5868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62c8e6682f95424bbeab0e7e770b7106: Sending DeleteTablet for 3 replicas of tablet f7166cf9eea94c0aa5dac6a6ccb68a6a
I20260812 14:55:49.392804 6552 tablet_service.cc:1514] Processing DeleteTablet for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:49 UTC) from {username='slave'} at 127.0.0.1:32868
I20260812 14:55:49.392829 6078 tablet_service.cc:1514] Processing DeleteTablet for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:49 UTC) from {username='slave'} at 127.0.0.1:52232
I20260812 14:55:49.393055 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5977
I20260812 14:55:49.393121 6306 tablet_service.cc:1514] Processing DeleteTablet for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:49 UTC) from {username='slave'} at 127.0.0.1:40574
I20260812 14:55:49.410205 6741 tablet_replica.cc:330] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: stopping tablet replica
I20260812 14:55:49.410434 6741 raft_consensus.cc:2229] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:55:49.410599 6741 raft_consensus.cc:2258] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:49.414606 6741 ts_tablet_manager.cc:1905] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:49.415969 6739 tablet_replica.cc:330] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: stopping tablet replica
I20260812 14:55:49.416100 6739 raft_consensus.cc:2229] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:49.416236 6739 raft_consensus.cc:2258] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:49.416440 6741 ts_tablet_manager.cc:1918] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:49.416512 6741 log.cc:1196] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/f7166cf9eea94c0aa5dac6a6ccb68a6a
I20260812 14:55:49.416771 6741 ts_tablet_manager.cc:1939] T f7166cf9eea94c0aa5dac6a6ccb68a6a P e404afe42de54a309a6db10e16aa2eef: Deleting consensus metadata
I20260812 14:55:49.416931 6739 ts_tablet_manager.cc:1905] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:49.417204 5851 catalog_manager.cc:4841] TS e404afe42de54a309a6db10e16aa2eef (127.4.27.2:36667): tablet f7166cf9eea94c0aa5dac6a6ccb68a6a (table test-table [id=ac6df5e0f5494a019d46b8f44636bd27]) successfully deleted
I20260812 14:55:49.418320 6739 ts_tablet_manager.cc:1918] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:49.418407 6739 log.cc:1196] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/f7166cf9eea94c0aa5dac6a6ccb68a6a
I20260812 14:55:49.418607 6739 ts_tablet_manager.cc:1939] T f7166cf9eea94c0aa5dac6a6ccb68a6a P 0349a0dd2d0f4e14a5791eafef5aa023: Deleting consensus metadata
I20260812 14:55:49.418977 5849 catalog_manager.cc:4841] TS 0349a0dd2d0f4e14a5791eafef5aa023 (127.4.27.3:38509): tablet f7166cf9eea94c0aa5dac6a6ccb68a6a (table test-table [id=ac6df5e0f5494a019d46b8f44636bd27]) successfully deleted
W20260812 14:55:49.419342 5851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:45727 (error 108)
I20260812 14:55:49.419636 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6205
I20260812 14:55:49.419773 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:49.419652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:45727 (local address 127.0.0.1:52238)
0812 14:55:49.419699 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:49.419743 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:45727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:55:49.432929 5851 catalog_manager.cc:4584] TS d2a697b1d73c49c581556f06c08ff9f9 (127.4.27.1:45727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7166cf9eea94c0aa5dac6a6ccb68a6a: Network error: Client connection negotiation failed: client connection to 127.4.27.1:45727: connect: Connection refused (error 111)
I20260812 14:55:49.443164 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6439
I20260812 14:55:49.458742 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 5816
2026-08-12T14:55:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (826 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:49 dist-test-slave-6zbq krb5kdc[6750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:49 dist-test-slave-6zbq krb5kdc[6750](info): set up 2 sockets
Aug 12 14:55:49 dist-test-slave-6zbq krb5kdc[6750](info): commencing operation
krb5kdc: starting...
W20260812 14:55:50.654307 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:50 dist-test-slave-6zbq krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:55:50.771030 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46271
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:50.850612 6776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:50.850802 6776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:50.850833 6776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:50.851928 6776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:50.851970 6776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:50.851989 6776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:50.852005 6776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:50.852022 6776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:50.853472 6776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:50.853700 6776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:50.853891 6776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:50.858652 6785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:50.858661 6791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:50.858712 6789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:50.858990 6776 server_base.cc:1034] running on GCE node
I20260812 14:55:50.859318 6776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:50.859608 6776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:50.860761 6776 hybrid_clock.cc:648] HybridClock initialized: now 1786546550860746 us; error 41 us; skew 500 ppm
Aug 12 14:55:50 dist-test-slave-6zbq krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546550, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:50.863268 6776 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:55:50.863648 6776 webserver.cc:460] Webserver started at http://127.4.27.62:40537/ using document root <none> and password file <none>
I20260812 14:55:50.863835 6776 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:50.863883 6776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:50.864002 6776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:50.865042 6776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "9abbfa11390444c8a46aae2d8f11e5cd"
format_stamp: "Formatted at 2026-08-12 14:55:50 on dist-test-slave-6zbq"
server_key: "35534a1a75632b19986ffa897e41e606"
server_key_iv: "b2d9615955a437b471ce0db889cc1b0d"
server_key_version: "encryptionkey@0"
I20260812 14:55:50.865381 6776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "9abbfa11390444c8a46aae2d8f11e5cd"
format_stamp: "Formatted at 2026-08-12 14:55:50 on dist-test-slave-6zbq"
server_key: "35534a1a75632b19986ffa897e41e606"
server_key_iv: "b2d9615955a437b471ce0db889cc1b0d"
server_key_version: "encryptionkey@0"
I20260812 14:55:50.866629 6776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:55:50.867277 6802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:50.867424 6776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:50.867507 6776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "9abbfa11390444c8a46aae2d8f11e5cd"
format_stamp: "Formatted at 2026-08-12 14:55:50 on dist-test-slave-6zbq"
server_key: "35534a1a75632b19986ffa897e41e606"
server_key_iv: "b2d9615955a437b471ce0db889cc1b0d"
server_key_version: "encryptionkey@0"
I20260812 14:55:50.867563 6776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:50.873535 6776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:50.875447 6776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:50.875569 6776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:50.879205 6776 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46271
I20260812 14:55:50.879236 6894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46271 every 8 connection(s)
I20260812 14:55:50.879519 6776 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:50.880287 6895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:50.882473 6895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd: Bootstrap starting.
I20260812 14:55:50.883014 6895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:50.883282 6895 log.cc:824] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:50.883894 6895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd: No bootstrap required, opened a new log
I20260812 14:55:50.885171 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6776
I20260812 14:55:50.885308 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:50.885512 4204 external_mini_cluster.cc:1428] Setting key 1f7960305f490133b245d0a3546bcc2c
I20260812 14:55:50.885550 6895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46271 } }
I20260812 14:55:50.885710 6895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:50.885756 6895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abbfa11390444c8a46aae2d8f11e5cd, State: Initialized, Role: FOLLOWER
I20260812 14:55:50.885896 6895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46271 } }
I20260812 14:55:50.885983 6895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:50.886021 6895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:50.886062 6895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:50.886656 6895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46271 } }
I20260812 14:55:50.886746 6895 leader_election.cc:302] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abbfa11390444c8a46aae2d8f11e5cd; no voters:
I20260812 14:55:50.886900 6895 leader_election.cc:288] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:50.886940 6906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:50.887045 6906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [term 1 LEADER]: Becoming Leader. State: Replica: 9abbfa11390444c8a46aae2d8f11e5cd, State: Running, Role: LEADER
I20260812 14:55:50.887116 6906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46271 } }
I20260812 14:55:50.887138 6895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:50.887508 6907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46271 } } }
I20260812 14:55:50.887622 6907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:50.887749 6906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abbfa11390444c8a46aae2d8f11e5cd. Latest consensus state: current_term: 1 leader_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abbfa11390444c8a46aae2d8f11e5cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46271 } } }
I20260812 14:55:50.887842 6906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:50.888139 6916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:50.888784 6916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:50.891364 6916 catalog_manager.cc:1347] Generated new cluster ID: e4dbec3d74d847d888a901f0a0853bec
I20260812 14:55:50.891418 6916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:55:50 dist-test-slave-6zbq krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:50.896036 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:50.886105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.4.27.62:46271)
0812 14:55:50.886280 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:50.886287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:50.886310 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:55:50.886638 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:50.886644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:50.886695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:50.886826 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:50.887583 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:50.889227 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:50.889996 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:50.890484 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:50.890590 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:50.893037 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:50.893041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:50.893042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:50.893053 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:50.894489 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:50.894859 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:50.894862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:50.894863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:50.894986 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:50.895158 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:50.895159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:50.895159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:50.895467 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:50.895630 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:50.895699 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 14:55:50.895836 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260812 14:55:50.903059 6916 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:50.903769 6916 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:50.917749 6916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd: Generated new TSK 0
I20260812 14:55:50.917968 6916 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.27.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
Entry for principal kudu/127.4.27.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab.
I20260812 14:55:50.942790 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46271
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:51.025861 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:51.026054 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:51.026077 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:51.027230 6941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:51.027273 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:51.027298 6941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:51.028784 6941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.1.keytab
--principal=kudu/127.4.27.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:51.029018 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:51.029246 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:51.034099 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.034107 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.034265 6941 server_base.cc:1034] running on GCE node
W20260812 14:55:51.034160 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.034575 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:51.034807 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:51.035938 6941 hybrid_clock.cc:648] HybridClock initialized: now 1786546551035927 us; error 29 us; skew 500 ppm
Aug 12 14:55:51 dist-test-slave-6zbq krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546551, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:51.038309 6941 init.cc:377] Logged in from keytab as kudu/127.4.27.1@KRBTEST.COM (short username kudu)
I20260812 14:55:51.038723 6941 webserver.cc:460] Webserver started at http://127.4.27.1:37947/ using document root <none> and password file <none>
I20260812 14:55:51.038924 6941 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:51.038970 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:51.039093 6941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:51.040091 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "0523df158a174305815875e18a5bc1a8"
server_key_iv: "ea6c2b50bde1220cbd3a8e6574597c49"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.040426 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "0523df158a174305815875e18a5bc1a8"
server_key_iv: "ea6c2b50bde1220cbd3a8e6574597c49"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.041894 6941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:55:51.042650 6965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.042799 6941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:51.042884 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "0523df158a174305815875e18a5bc1a8"
server_key_iv: "ea6c2b50bde1220cbd3a8e6574597c49"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.042945 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:51.056926 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:51.058629 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:51.058764 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:51.058969 6941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:51.059247 6941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:51.059280 6941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.059309 6941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:51.059333 6941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.064730 6941 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:39993
I20260812 14:55:51.064783 7153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:39993 every 8 connection(s)
I20260812 14:55:51.065048 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:51.067324 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 6941
I20260812 14:55:51.067452 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:55:51.067660 4204 external_mini_cluster.cc:1428] Setting key 2f09f53fa03d692fab725fcba071eb82
Aug 12 14:55:51 dist-test-slave-6zbq krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546551, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.1@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:51.074398 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.065710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:50243 (local address 127.4.27.62:46271)
0812 14:55:51.065788 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.065791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.065934 (+ 143us) server_negotiation.cc:407] Connection header received
0812 14:55:51.066434 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.066435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.066445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.066490 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:51.067494 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.068087 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.069073 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.069284 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.069342 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.071668 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:51.071670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:51.071670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:51.071677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:51.072939 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.073388 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.073388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.073389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.073435 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.073685 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.073685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.073686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.073778 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:51.073824 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.074199 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.074291 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:55:51.074553 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.065668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46271 (local address 127.4.27.1:50243)
0812 14:55:51.065823 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.065858 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.066323 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.066551 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.066554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.067097 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:51.067370 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.067376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.068188 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.068190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.068987 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.068990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.069356 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.069357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.069478 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.069980 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:51.069991 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:51.071582 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:51.073017 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:51.073025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:51.073027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:51.073317 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:51.073490 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:51.073491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:51.073491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:51.073627 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:51.073836 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:51.073842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:51.074006 (+ 164us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.074129 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.074300 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260812 14:55:51.075057 7156 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46271
I20260812 14:55:51.075160 7156 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:51.075330 7156 heartbeater.cc:511] Master 127.4.27.62:46271 requested a full tablet report, sending...
I20260812 14:55:51.075855 6832 ts_manager.cc:194] Registered new tserver with Master: ff67b2a3fc9d4f7394a0596a17dadd0e (127.4.27.1:39993)
I20260812 14:55:51.076719 6832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.1@KRBTEST.COM'} at 127.4.27.1:50243
WARNING: no policy specified for kudu/127.4.27.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
Entry for principal kudu/127.4.27.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab.
I20260812 14:55:51.093083 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46271
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:51.173955 7166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:51.174134 7166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:51.174154 7166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:51.175271 7166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:51.175310 7166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:51.175336 7166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:51.176729 7166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.2.keytab
--principal=kudu/127.4.27.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:51.176927 7166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:51.177160 7166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:51.181866 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.181875 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.181875 7187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.182035 7166 server_base.cc:1034] running on GCE node
I20260812 14:55:51.182235 7166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:51.182476 7166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:51.183616 7166 hybrid_clock.cc:648] HybridClock initialized: now 1786546551183606 us; error 29 us; skew 500 ppm
Aug 12 14:55:51 dist-test-slave-6zbq krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546551, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:51.185916 7166 init.cc:377] Logged in from keytab as kudu/127.4.27.2@KRBTEST.COM (short username kudu)
I20260812 14:55:51.186291 7166 webserver.cc:460] Webserver started at http://127.4.27.2:33875/ using document root <none> and password file <none>
I20260812 14:55:51.186539 7166 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:51.186585 7166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:51.186683 7166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:51.187706 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "a037d81dda1442f3b0215b33bdaacdeb"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "6e952e6d1dd841cd88d185e530d5899d"
server_key_iv: "b527fffee5cf3d356436a68e38a95cbe"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.188017 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "a037d81dda1442f3b0215b33bdaacdeb"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "6e952e6d1dd841cd88d185e530d5899d"
server_key_iv: "b527fffee5cf3d356436a68e38a95cbe"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.189325 7166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:51.190099 7197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.190407 7166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:55:51.190497 7166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "a037d81dda1442f3b0215b33bdaacdeb"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "6e952e6d1dd841cd88d185e530d5899d"
server_key_iv: "b527fffee5cf3d356436a68e38a95cbe"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.190551 7166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:51.202198 7166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:51.203781 7166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:51.203881 7166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:51.204043 7166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:51.204277 7166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:51.204306 7166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.204331 7166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:51.204346 7166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.209574 7166 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:42775
I20260812 14:55:51.209637 7401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:42775 every 8 connection(s)
I20260812 14:55:51.209875 7166 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
Aug 12 14:55:51 dist-test-slave-6zbq krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546551, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.2@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:51.217682 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7166
I20260812 14:55:51.217795 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:55:51.217936 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.210527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:58179 (local address 127.4.27.62:46271)
0812 14:55:51.210575 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.210577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.210748 (+ 171us) server_negotiation.cc:407] Connection header received
0812 14:55:51.211233 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.211235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.211247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.211298 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:51.212172 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.212663 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.213598 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.213791 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.213848 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.215719 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:51.215721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:51.215721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:51.215730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:51.216767 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.217100 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.217101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.217102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.217153 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.217352 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.217353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.217353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.217458 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:51.217515 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.217772 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.217851 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:55:51.217967 4204 external_mini_cluster.cc:1428] Setting key 44bf044737f26be7a2fbafcf1affa3b7
I20260812 14:55:51.218142 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.210500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46271 (local address 127.4.27.2:58179)
0812 14:55:51.210630 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.210668 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.211132 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.211364 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.211367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.211900 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:51.212073 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.212077 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.212742 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.212743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.213511 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.213513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.213861 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.213862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.213984 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.214412 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:51.214420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:51.215636 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:51.216836 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:51.216842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:51.216844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:51.217037 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:51.217206 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:51.217207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:51.217207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:51.217297 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:51.217532 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:51.217538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:51.217657 (+ 119us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.217741 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.217878 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260812 14:55:51.218689 7402 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46271
I20260812 14:55:51.218798 7402 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:51.218966 7402 heartbeater.cc:511] Master 127.4.27.62:46271 requested a full tablet report, sending...
I20260812 14:55:51.219339 6831 ts_manager.cc:194] Registered new tserver with Master: a037d81dda1442f3b0215b33bdaacdeb (127.4.27.2:42775)
I20260812 14:55:51.219779 6831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.2@KRBTEST.COM'} at 127.4.27.2:58179
WARNING: no policy specified for kudu/127.4.27.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
Entry for principal kudu/127.4.27.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab.
I20260812 14:55:51.240648 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46271
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:51.323990 7411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:51.324170 7411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:51.324193 7411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:51.325286 7411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:51.325328 7411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:51.325353 7411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:51.326795 7411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.3.keytab
--principal=kudu/127.4.27.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:46271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:51.326973 7411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:51.327292 7411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:51.332060 7422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.332060 7420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.332073 7419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.332223 7411 server_base.cc:1034] running on GCE node
I20260812 14:55:51.332429 7411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:51.332645 7411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:51.333787 7411 hybrid_clock.cc:648] HybridClock initialized: now 1786546551333772 us; error 27 us; skew 500 ppm
Aug 12 14:55:51 dist-test-slave-6zbq krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546551, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:51.336208 7411 init.cc:377] Logged in from keytab as kudu/127.4.27.3@KRBTEST.COM (short username kudu)
I20260812 14:55:51.336678 7411 webserver.cc:460] Webserver started at http://127.4.27.3:40593/ using document root <none> and password file <none>
I20260812 14:55:51.336871 7411 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:51.336920 7411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:51.337042 7411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:51.338024 7411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "4fe0e6810ec14717b08db65001cf5d81"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "1aa0f96638222d363ebe4756a82e4aed"
server_key_iv: "acdaf7b492aba8356e5161503a2013cc"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.338374 7411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "4fe0e6810ec14717b08db65001cf5d81"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "1aa0f96638222d363ebe4756a82e4aed"
server_key_iv: "acdaf7b492aba8356e5161503a2013cc"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.339563 7411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:55:51.340199 7434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.340349 7411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:51.340432 7411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "4fe0e6810ec14717b08db65001cf5d81"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "1aa0f96638222d363ebe4756a82e4aed"
server_key_iv: "acdaf7b492aba8356e5161503a2013cc"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.340495 7411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:51.351421 7411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:51.352957 7411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:51.353081 7411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:51.353266 7411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:51.353536 7411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:51.353569 7411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.353595 7411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:51.353621 7411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.359368 7411 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:46307
I20260812 14:55:51.359424 7634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:46307 every 8 connection(s)
I20260812 14:55:51.359695 7411 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:55:51.364892 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7411
Aug 12 14:55:51 dist-test-slave-6zbq krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546551, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.3@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:55:51.365002 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:55:51.365193 4204 external_mini_cluster.cc:1428] Setting key 308ad34c1208071c14946d7c820460c7
I20260812 14:55:51.367990 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.360323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:49247 (local address 127.4.27.62:46271)
0812 14:55:51.360383 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.360385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.360563 (+ 178us) server_negotiation.cc:407] Connection header received
0812 14:55:51.361003 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.361004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.361014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.361063 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:51.361917 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.362430 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.363323 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.363496 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.363537 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.365464 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:51.365466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:51.365466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:51.365474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:51.366805 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.367147 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.367148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.367148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.367206 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.367414 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.367415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.367415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.367525 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:51.367581 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.367825 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.367906 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 14:55:51.368129 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.360306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:46271 (local address 127.4.27.3:49247)
0812 14:55:51.360454 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.360487 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.360932 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.361127 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.361130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.361656 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:51.361844 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.361848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.362488 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.362489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.363263 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.363265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.363548 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.363548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.363637 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.364022 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:55:51.364030 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:51.365378 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:51.366876 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:51.366882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:51.366884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:51.367083 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:51.367260 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:55:51.367260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:55:51.367261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:55:51.367359 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:55:51.367579 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:51.367586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:51.367710 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.367794 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.367937 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260812 14:55:51.368549 7635 heartbeater.cc:347] Connected to a master server at 127.4.27.62:46271
I20260812 14:55:51.368637 7635 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:51.368798 7635 heartbeater.cc:511] Master 127.4.27.62:46271 requested a full tablet report, sending...
I20260812 14:55:51.369125 6831 ts_manager.cc:194] Registered new tserver with Master: 4fe0e6810ec14717b08db65001cf5d81 (127.4.27.3:46307)
I20260812 14:55:51.369448 6831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.27.3@KRBTEST.COM'} at 127.4.27.3:49247
I20260812 14:55:51.376734 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:51.382911 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.377996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.4.27.62:46271)
0812 14:55:51.378032 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.378034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.378134 (+ 100us) server_negotiation.cc:407] Connection header received
0812 14:55:51.378179 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.378180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.378187 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.378226 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:51.378843 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.379275 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.379912 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.380074 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.380114 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.381133 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:51.381134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:51.381134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:51.381138 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:51.382096 (+ 958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.382399 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.382400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.382400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.382442 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:51.382561 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:51.382562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:51.382562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:51.382665 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:51.382742 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.382803 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.382859 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:55:51.384370 6831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:51.385833 6831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:51.391505 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.387701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:46307 (local address 127.0.0.1:56568)
0812 14:55:51.388135 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.388145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.388196 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.388348 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.388349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.388351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:51.388454 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.388457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.389365 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.389366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.390444 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.390447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.391234 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.391311 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.391323 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.391386 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.391438 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":60,"threads_started":1}
I20260812 14:55:51.391511 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.387886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.4.27.3:46307)
0812 14:55:51.388048 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.388052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.388193 (+ 141us) server_negotiation.cc:407] Connection header received
0812 14:55:51.388257 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.388259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.388274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.388338 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:51.388529 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.389276 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.390536 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.391162 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391228 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.391282 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.391405 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.391438 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260812 14:55:51.391590 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.387691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:39993 (local address 127.0.0.1:35740)
0812 14:55:51.387845 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.387878 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.388041 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.388341 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.388344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.388346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:51.388405 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.388407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.389721 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.389722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.390566 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.390569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.391365 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.391429 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.391433 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.391478 (+ 45us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.391520 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":43,"threads_started":1}
I20260812 14:55:51.391817 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.387795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.4.27.1:39993)
0812 14:55:51.387975 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.387979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.387991 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:55:51.388156 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.388158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.388175 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.388250 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:51.388491 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.389596 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.390630 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.391213 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391280 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.391330 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.391697 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.391741 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260812 14:55:51.392011 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.387697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:42775 (local address 127.0.0.1:59446)
0812 14:55:51.388033 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.388043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.388096 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.388382 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.388383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.388385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:51.388422 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.388426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.389856 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.389857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.390924 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.390927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.391790 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.391856 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.391859 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.391908 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.391945 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":47,"threads_started":1}
I20260812 14:55:51.392230 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.387914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.4.27.2:42775)
0812 14:55:51.388080 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.388085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.388096 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:55:51.388207 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.388210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.388224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.388307 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:51.388497 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.389760 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391010 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.391711 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.391781 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.391832 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.392117 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.392160 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260812 14:55:51.392794 7285 tablet_service.cc:1467] Processing CreateTablet for tablet 42a813ca2c0c46d2967b2a023651ce6b (DEFAULT_TABLE table=test-table [id=719d45e0b157495193513e87bf4d5f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:51.392745 7046 tablet_service.cc:1467] Processing CreateTablet for tablet 42a813ca2c0c46d2967b2a023651ce6b (DEFAULT_TABLE table=test-table [id=719d45e0b157495193513e87bf4d5f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:51.393146 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a813ca2c0c46d2967b2a023651ce6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:51.393146 7285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a813ca2c0c46d2967b2a023651ce6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:51.396323 7669 tablet_bootstrap.cc:492] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: Bootstrap starting.
I20260812 14:55:51.396677 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 42a813ca2c0c46d2967b2a023651ce6b (DEFAULT_TABLE table=test-table [id=719d45e0b157495193513e87bf4d5f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:51.396889 7669 tablet_bootstrap.cc:654] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:51.396947 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42a813ca2c0c46d2967b2a023651ce6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:51.397396 7669 log.cc:824] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:51.397712 7663 tablet_bootstrap.cc:492] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Bootstrap starting.
I20260812 14:55:51.398205 7663 tablet_bootstrap.cc:654] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:51.398663 7663 log.cc:824] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:51.399183 7669 tablet_bootstrap.cc:492] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: No bootstrap required, opened a new log
I20260812 14:55:51.399255 7669 ts_tablet_manager.cc:1397] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 14:55:51.399364 7663 tablet_bootstrap.cc:492] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: No bootstrap required, opened a new log
I20260812 14:55:51.399422 7663 ts_tablet_manager.cc:1397] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 14:55:51.399750 7676 tablet_bootstrap.cc:492] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Bootstrap starting.
I20260812 14:55:51.400233 7676 tablet_bootstrap.cc:654] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:51.400678 7676 log.cc:824] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:51.401203 7663 raft_consensus.cc:348] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.401167 7669 raft_consensus.cc:348] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.401319 7676 tablet_bootstrap.cc:492] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: No bootstrap required, opened a new log
I20260812 14:55:51.401357 7669 raft_consensus.cc:374] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:51.401357 7663 raft_consensus.cc:374] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:51.401379 7676 ts_tablet_manager.cc:1397] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:51.401394 7663 raft_consensus.cc:729] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a037d81dda1442f3b0215b33bdaacdeb, State: Initialized, Role: FOLLOWER
I20260812 14:55:51.401394 7669 raft_consensus.cc:729] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff67b2a3fc9d4f7394a0596a17dadd0e, State: Initialized, Role: FOLLOWER
I20260812 14:55:51.401510 7663 consensus_queue.cc:260] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.401530 7669 consensus_queue.cc:260] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.401784 7669 ts_tablet_manager.cc:1428] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:55:51.401783 7663 ts_tablet_manager.cc:1428] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 14:55:51.401780 7402 heartbeater.cc:503] Master 127.4.27.62:46271 was elected leader, sending a full tablet report...
I20260812 14:55:51.401908 7156 heartbeater.cc:503] Master 127.4.27.62:46271 was elected leader, sending a full tablet report...
I20260812 14:55:51.403136 7676 raft_consensus.cc:348] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.403258 7676 raft_consensus.cc:374] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:51.403288 7676 raft_consensus.cc:729] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe0e6810ec14717b08db65001cf5d81, State: Initialized, Role: FOLLOWER
I20260812 14:55:51.403400 7676 consensus_queue.cc:260] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.403628 7676 ts_tablet_manager.cc:1428] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 14:55:51.403656 7635 heartbeater.cc:503] Master 127.4.27.62:46271 was elected leader, sending a full tablet report...
W20260812 14:55:51.460382 7403 tablet.cc:2369] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:51.528942 7686 raft_consensus.cc:482] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:51.529080 7686 raft_consensus.cc:504] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.529444 7686 leader_election.cc:288] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff67b2a3fc9d4f7394a0596a17dadd0e (127.4.27.1:39993), a037d81dda1442f3b0215b33bdaacdeb (127.4.27.2:42775)
I20260812 14:55:51.533206 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.529726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:34913 (local address 127.4.27.1:39993)
0812 14:55:51.529772 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.529774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.529783 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:55:51.529812 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.529813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.529825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.529879 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:51.530057 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.530836 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.532248 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.532854 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.532930 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.532940 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.533093 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.533131 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260812 14:55:51.533203 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.529735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:45321 (local address 127.4.27.2:42775)
0812 14:55:51.529779 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.529781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.529910 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:55:51.529945 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.529946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.529959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.530000 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:51.530132 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.530852 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.532249 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.532854 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.532930 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.532940 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.533095 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.533131 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:55:51.533217 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.529617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:39993 (local address 127.4.27.3:34913)
0812 14:55:51.529677 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.529690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.529740 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.529907 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.529908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.529910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:51.529985 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.529988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.530912 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.530912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.532142 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.532146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.532935 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.532936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.533024 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.533026 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.533081 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.533122 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":17}
I20260812 14:55:51.533236 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.529618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:42775 (local address 127.4.27.3:45321)
0812 14:55:51.529844 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.529851 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.529907 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.530013 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.530014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.530015 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:51.530077 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.530078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.530912 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.530912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.532142 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.532146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.532935 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.532936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.533021 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.533026 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.533075 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.533122 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":122,"thread_start_us":113,"threads_started":1}
I20260812 14:55:51.533550 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a813ca2c0c46d2967b2a023651ce6b" candidate_uuid: "4fe0e6810ec14717b08db65001cf5d81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" is_pre_election: true
I20260812 14:55:51.533701 7082 raft_consensus.cc:2454] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe0e6810ec14717b08db65001cf5d81 in term 0.
I20260812 14:55:51.533811 7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a813ca2c0c46d2967b2a023651ce6b" candidate_uuid: "4fe0e6810ec14717b08db65001cf5d81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a037d81dda1442f3b0215b33bdaacdeb" is_pre_election: true
I20260812 14:55:51.533926 7439 leader_election.cc:302] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0e6810ec14717b08db65001cf5d81, ff67b2a3fc9d4f7394a0596a17dadd0e; no voters:
I20260812 14:55:51.533967 7326 raft_consensus.cc:2454] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe0e6810ec14717b08db65001cf5d81 in term 0.
I20260812 14:55:51.534085 7686 raft_consensus.cc:2781] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:51.534142 7686 raft_consensus.cc:482] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:51.534161 7686 raft_consensus.cc:3037] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:51.534890 7686 raft_consensus.cc:504] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.535017 7686 leader_election.cc:288] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [CANDIDATE]: Term 1 election: Requested vote from peers ff67b2a3fc9d4f7394a0596a17dadd0e (127.4.27.1:39993), a037d81dda1442f3b0215b33bdaacdeb (127.4.27.2:42775)
I20260812 14:55:51.535166 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a813ca2c0c46d2967b2a023651ce6b" candidate_uuid: "4fe0e6810ec14717b08db65001cf5d81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e"
I20260812 14:55:51.535187 7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42a813ca2c0c46d2967b2a023651ce6b" candidate_uuid: "4fe0e6810ec14717b08db65001cf5d81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a037d81dda1442f3b0215b33bdaacdeb"
I20260812 14:55:51.535243 7082 raft_consensus.cc:3037] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:51.535264 7326 raft_consensus.cc:3037] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:51.535899 7082 raft_consensus.cc:2454] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe0e6810ec14717b08db65001cf5d81 in term 1.
I20260812 14:55:51.536008 7326 raft_consensus.cc:2454] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe0e6810ec14717b08db65001cf5d81 in term 1.
I20260812 14:55:51.536051 7439 leader_election.cc:302] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe0e6810ec14717b08db65001cf5d81, ff67b2a3fc9d4f7394a0596a17dadd0e; no voters:
I20260812 14:55:51.536164 7686 raft_consensus.cc:2781] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:51.536314 7686 raft_consensus.cc:686] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe0e6810ec14717b08db65001cf5d81, State: Running, Role: LEADER
I20260812 14:55:51.536401 7686 consensus_queue.cc:237] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } }
I20260812 14:55:51.537194 6831 catalog_manager.cc:5495] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe0e6810ec14717b08db65001cf5d81 (127.4.27.3). New cstate: current_term: 1 leader_uuid: "4fe0e6810ec14717b08db65001cf5d81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe0e6810ec14717b08db65001cf5d81" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 } health_report { overall_health: UNKNOWN } } }
W20260812 14:55:51.565670 7157 tablet.cc:2369] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:51.571666 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.569453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.4.27.3:46307)
0812 14:55:51.569494 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.569496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.569505 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:55:51.569540 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.569541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.569559 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.569624 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:51.569730 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.570171 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.570977 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.571162 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.571218 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.571368 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:51.571463 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:51.571533 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.571572 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.571595 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:55:51.573328 7082 raft_consensus.cc:1261] T 42a813ca2c0c46d2967b2a023651ce6b P ff67b2a3fc9d4f7394a0596a17dadd0e [term 1 FOLLOWER]: Refusing update from remote peer 4fe0e6810ec14717b08db65001cf5d81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:51.573344 7326 raft_consensus.cc:1261] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 1 FOLLOWER]: Refusing update from remote peer 4fe0e6810ec14717b08db65001cf5d81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:51.573663 7688 consensus_queue.cc:1035] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a037d81dda1442f3b0215b33bdaacdeb" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:51.573799 7686 consensus_queue.cc:1035] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff67b2a3fc9d4f7394a0596a17dadd0e" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:51.576004 7700 mvcc.cc:205] Tried to move back new op lower bound from 7317694675242020864 to 7317694675093680128. Current Snapshot: MvccSnapshot[applied={T|T < 7317694675242020864}]
I20260812 14:55:51.576251 7702 mvcc.cc:205] Tried to move back new op lower bound from 7317694675242020864 to 7317694675093680128. Current Snapshot: MvccSnapshot[applied={T|T < 7317694675242020864}]
I20260812 14:55:51.576452 7701 mvcc.cc:205] Tried to move back new op lower bound from 7317694675242020864 to 7317694675093680128. Current Snapshot: MvccSnapshot[applied={T|T < 7317694675242020864}]
I20260812 14:55:51.578277 6831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:51.578380 6831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:51.579061 6831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9abbfa11390444c8a46aae2d8f11e5cd: Sending DeleteTablet for 3 replicas of tablet 42a813ca2c0c46d2967b2a023651ce6b
I20260812 14:55:51.579389 7285 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a813ca2c0c46d2967b2a023651ce6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:51 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:59446
I20260812 14:55:51.579387 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a813ca2c0c46d2967b2a023651ce6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:51 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:56568
I20260812 14:55:51.579397 7046 tablet_service.cc:1514] Processing DeleteTablet for tablet 42a813ca2c0c46d2967b2a023651ce6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:51 UTC) from {username='kudu', principal='kudu/127.4.27.62@KRBTEST.COM'} at 127.0.0.1:35740
I20260812 14:55:51.579615 7714 tablet_replica.cc:330] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: stopping tablet replica
I20260812 14:55:51.579722 7714 raft_consensus.cc:2229] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:55:51.579818 7714 raft_consensus.cc:2258] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:51.579823 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6941
I20260812 14:55:51.583004 7714 ts_tablet_manager.cc:1905] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:51.584627 7714 ts_tablet_manager.cc:1918] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:51.584689 7714 log.cc:1196] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/42a813ca2c0c46d2967b2a023651ce6b
I20260812 14:55:51.584968 7714 ts_tablet_manager.cc:1939] T 42a813ca2c0c46d2967b2a023651ce6b P 4fe0e6810ec14717b08db65001cf5d81: Deleting consensus metadata
I20260812 14:55:51.600459 7715 tablet_replica.cc:330] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: stopping tablet replica
I20260812 14:55:51.600579 7715 raft_consensus.cc:2229] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:51.600687 7715 raft_consensus.cc:2258] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:51.600749 6804 catalog_manager.cc:4841] TS 4fe0e6810ec14717b08db65001cf5d81 (127.4.27.3:46307): tablet 42a813ca2c0c46d2967b2a023651ce6b (table test-table [id=719d45e0b157495193513e87bf4d5f5b]) successfully deleted
I20260812 14:55:51.601387 7715 ts_tablet_manager.cc:1905] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:51.602948 7715 ts_tablet_manager.cc:1918] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:51.603021 7715 log.cc:1196] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/42a813ca2c0c46d2967b2a023651ce6b
I20260812 14:55:51.603312 7715 ts_tablet_manager.cc:1939] T 42a813ca2c0c46d2967b2a023651ce6b P a037d81dda1442f3b0215b33bdaacdeb: Deleting consensus metadata
I20260812 14:55:51.603716 6808 catalog_manager.cc:4841] TS a037d81dda1442f3b0215b33bdaacdeb (127.4.27.2:42775): tablet 42a813ca2c0c46d2967b2a023651ce6b (table test-table [id=719d45e0b157495193513e87bf4d5f5b]) successfully deleted
W20260812 14:55:51.604403 6807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:39993 (error 108)
I20260812 14:55:51.604641 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7166
I20260812 14:55:51.609222 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.604759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:39993 (local address 127.0.0.1:35756)
0812 14:55:51.609138 (+ 4379us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.609189 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4354}
W20260812 14:55:51.611083 6807 catalog_manager.cc:4584] TS ff67b2a3fc9d4f7394a0596a17dadd0e (127.4.27.1:39993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42a813ca2c0c46d2967b2a023651ce6b: Network error: Client connection negotiation failed: client connection to 127.4.27.1:39993: connect: Connection refused (error 111)
I20260812 14:55:51.625053 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7411
I20260812 14:55:51.636511 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 6776
2026-08-12T14:55:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2179 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T14:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:51Z Disabled control of system clock
I20260812 14:55:51.668342 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41111
--webserver_interface=127.4.27.62
--webserver_port=0
--builtin_ntp_servers=127.4.27.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:51.747084 7734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:51.747292 7734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:51.747311 7734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:51.748456 7734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:51.748497 7734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:51.748508 7734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:51.748519 7734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:51.748530 7734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:51.749853 7734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:41111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:51.750073 7734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:51.750334 7734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:51.755003 7743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.755004 7747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.755028 7745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.755138 7734 server_base.cc:1034] running on GCE node
I20260812 14:55:51.755474 7734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:51.755741 7734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:51.756877 7734 hybrid_clock.cc:648] HybridClock initialized: now 1786546551756865 us; error 28 us; skew 500 ppm
I20260812 14:55:51.758045 7734 webserver.cc:460] Webserver started at http://127.4.27.62:45713/ using document root <none> and password file <none>
I20260812 14:55:51.758251 7734 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:51.758301 7734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:51.758447 7734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:51.759429 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "429f080089ad4f1fb6f30cf63be9a732"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "74967c4ceddeb7536da03e59afa286bc"
server_key_iv: "efb34a8cce1bbf456cd67c4ef923a44b"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.759747 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "429f080089ad4f1fb6f30cf63be9a732"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "74967c4ceddeb7536da03e59afa286bc"
server_key_iv: "efb34a8cce1bbf456cd67c4ef923a44b"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.760912 7734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:55:51.761525 7754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.761678 7734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:51.761760 7734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "429f080089ad4f1fb6f30cf63be9a732"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "74967c4ceddeb7536da03e59afa286bc"
server_key_iv: "efb34a8cce1bbf456cd67c4ef923a44b"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.761822 7734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:51.779683 7734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:51.779942 7734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:51.780063 7734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:51.783758 7734 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:41111
I20260812 14:55:51.783790 7838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:41111 every 8 connection(s)
I20260812 14:55:51.784080 7734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:51.784943 7839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:51.787138 7839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732: Bootstrap starting.
I20260812 14:55:51.787667 7839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:51.787890 7839 log.cc:824] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:51.788447 7839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732: No bootstrap required, opened a new log
I20260812 14:55:51.789899 7839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429f080089ad4f1fb6f30cf63be9a732" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41111 } }
I20260812 14:55:51.790023 7839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:51.790043 7839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429f080089ad4f1fb6f30cf63be9a732, State: Initialized, Role: FOLLOWER
I20260812 14:55:51.790149 7839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429f080089ad4f1fb6f30cf63be9a732" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41111 } }
I20260812 14:55:51.790225 7839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:51.790254 7839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:51.790285 7839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:51.790755 7839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429f080089ad4f1fb6f30cf63be9a732" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41111 } }
I20260812 14:55:51.790838 7839 leader_election.cc:302] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 429f080089ad4f1fb6f30cf63be9a732; no voters:
I20260812 14:55:51.790952 7839 leader_election.cc:288] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:51.791011 7845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:51.791155 7839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:51.791183 7845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [term 1 LEADER]: Becoming Leader. State: Replica: 429f080089ad4f1fb6f30cf63be9a732, State: Running, Role: LEADER
I20260812 14:55:51.791272 7845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429f080089ad4f1fb6f30cf63be9a732" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41111 } }
I20260812 14:55:51.791687 7846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "429f080089ad4f1fb6f30cf63be9a732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429f080089ad4f1fb6f30cf63be9a732" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41111 } } }
I20260812 14:55:51.791743 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429f080089ad4f1fb6f30cf63be9a732. Latest consensus state: current_term: 1 leader_uuid: "429f080089ad4f1fb6f30cf63be9a732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429f080089ad4f1fb6f30cf63be9a732" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41111 } } }
I20260812 14:55:51.791770 7846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:51.791788 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:51.792048 7864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:51.792539 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7734
I20260812 14:55:51.792599 7864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:51.792655 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:51.792829 4204 external_mini_cluster.cc:1428] Setting key 5ebc5666c7f49d79478a14738588ac96
I20260812 14:55:51.794293 7864 catalog_manager.cc:1347] Generated new cluster ID: 96b27d7040c547c8a4ac0a32f7dde4cf
I20260812 14:55:51.794351 7864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:51.799036 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.793351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.4.27.62:41111)
0812 14:55:51.793540 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.793548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.793569 (+ 21us) server_negotiation.cc:407] Connection header received
0812 14:55:51.793661 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.793666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.793724 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.793876 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:51.794123 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.795462 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.796593 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.797225 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.797348 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.798177 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:51.798181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:51.798182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:51.798429 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:51.798473 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:51.798487 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:51.798547 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.798754 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.798835 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260812 14:55:51.799481 7864 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:51.799968 7864 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:51.806288 7864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732: Generated new TSK 0
I20260812 14:55:51.806469 7864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:51.811030 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.1:0
--local_ip_for_outbound_sockets=127.4.27.1
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41111
--builtin_ntp_servers=127.4.27.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:51.891491 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:51.891674 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:51.891695 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:51.892869 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:51.892906 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:51.892937 7879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.1
I20260812 14:55:51.894238 7879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.27.1
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.1
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:51.894464 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:51.894737 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:51.899500 7887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.899479 7879 server_base.cc:1034] running on GCE node
W20260812 14:55:51.899461 7886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:51.899453 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:51.899874 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:51.900099 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:51.901242 7879 hybrid_clock.cc:648] HybridClock initialized: now 1786546551901230 us; error 29 us; skew 500 ppm
I20260812 14:55:51.902462 7879 webserver.cc:460] Webserver started at http://127.4.27.1:44681/ using document root <none> and password file <none>
I20260812 14:55:51.902671 7879 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:51.902717 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:51.902845 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:51.903821 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data/instance:
uuid: "43b943f3100544ac972585776a6cde31"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "c6b3e1dcb6c21067ece903b8c3ec07d3"
server_key_iv: "ae1c062b7da6b0093ad9d8ccf5e288c2"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.904143 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance:
uuid: "43b943f3100544ac972585776a6cde31"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "c6b3e1dcb6c21067ece903b8c3ec07d3"
server_key_iv: "ae1c062b7da6b0093ad9d8ccf5e288c2"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.905367 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:55:51.906090 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.906255 7879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:51.906360 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal
uuid: "43b943f3100544ac972585776a6cde31"
format_stamp: "Formatted at 2026-08-12 14:55:51 on dist-test-slave-6zbq"
server_key: "c6b3e1dcb6c21067ece903b8c3ec07d3"
server_key_iv: "ae1c062b7da6b0093ad9d8ccf5e288c2"
server_key_version: "encryptionkey@0"
I20260812 14:55:51.906433 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:51.913964 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:51.914194 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:51.914319 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:51.914533 7879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:51.914825 7879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:51.914860 7879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.914885 7879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:51.914909 7879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:51.920189 7879 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.1:41993
I20260812 14:55:51.920249 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.1:41993 every 8 connection(s)
I20260812 14:55:51.920507 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/data/info.pb
I20260812 14:55:51.925194 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.921128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.1:40457 (local address 127.4.27.62:41111)
0812 14:55:51.921196 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:51.921198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:51.921388 (+ 190us) server_negotiation.cc:407] Connection header received
0812 14:55:51.921866 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:51.921868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:51.921882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:51.921927 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:51.922602 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.923080 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.924015 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.924215 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.924271 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.924934 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:51.924935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:51.924935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:51.924940 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:51.924955 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:51.924959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:51.925004 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:51.925083 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 14:55:51.925111 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 14:55:51.925374 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:51.921123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41111 (local address 127.4.27.1:40457)
0812 14:55:51.921284 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:51.921314 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:51.921770 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:51.922006 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:51.922009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:51.922019 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:51.922517 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.922523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.923166 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.923168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.923926 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:51.923929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:51.924286 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:51.924287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:51.924420 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:51.924806 (+ 386us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:51.924806 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:51.924832 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:51.924853 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:51.925009 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:51.925014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:51.925017 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:51.925080 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:55:51.925178 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260812 14:55:51.925815 8113 heartbeater.cc:347] Connected to a master server at 127.4.27.62:41111
I20260812 14:55:51.925902 8113 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:51.926033 8113 heartbeater.cc:511] Master 127.4.27.62:41111 requested a full tablet report, sending...
I20260812 14:55:51.926569 7782 ts_manager.cc:194] Registered new tserver with Master: 43b943f3100544ac972585776a6cde31 (127.4.27.1:41993)
I20260812 14:55:51.927029 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 7879
I20260812 14:55:51.927160 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-0/wal/instance
I20260812 14:55:51.927346 4204 external_mini_cluster.cc:1428] Setting key ec99cbf69ce83a4dc6c32992e9c62df9
I20260812 14:55:51.927418 7782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.1:40457
I20260812 14:55:51.928368 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.2:0
--local_ip_for_outbound_sockets=127.4.27.2
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41111
--builtin_ntp_servers=127.4.27.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:52.007834 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:52.008020 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:52.008049 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:52.009196 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:52.009235 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:52.009266 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.2
I20260812 14:55:52.010630 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.27.2
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.2
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:52.010862 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:52.011118 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:52.015885 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:52.015878 8130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:52.016029 8118 server_base.cc:1034] running on GCE node
W20260812 14:55:52.015878 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:52.016278 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:52.016508 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:52.017654 8118 hybrid_clock.cc:648] HybridClock initialized: now 1786546552017637 us; error 29 us; skew 500 ppm
I20260812 14:55:52.018893 8118 webserver.cc:460] Webserver started at http://127.4.27.2:33971/ using document root <none> and password file <none>
I20260812 14:55:52.019114 8118 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:52.019164 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:52.019294 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:52.020260 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data/instance:
uuid: "5b275ee4c8ec46f68cfa47c293848405"
format_stamp: "Formatted at 2026-08-12 14:55:52 on dist-test-slave-6zbq"
server_key: "bea73d276ba5978ca2cbb6f63c01bec5"
server_key_iv: "16269c4a20ff8cc110ad422f6c5ee5e4"
server_key_version: "encryptionkey@0"
I20260812 14:55:52.020576 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance:
uuid: "5b275ee4c8ec46f68cfa47c293848405"
format_stamp: "Formatted at 2026-08-12 14:55:52 on dist-test-slave-6zbq"
server_key: "bea73d276ba5978ca2cbb6f63c01bec5"
server_key_iv: "16269c4a20ff8cc110ad422f6c5ee5e4"
server_key_version: "encryptionkey@0"
I20260812 14:55:52.021718 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:55:52.022317 8139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.022465 8118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.022545 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal
uuid: "5b275ee4c8ec46f68cfa47c293848405"
format_stamp: "Formatted at 2026-08-12 14:55:52 on dist-test-slave-6zbq"
server_key: "bea73d276ba5978ca2cbb6f63c01bec5"
server_key_iv: "16269c4a20ff8cc110ad422f6c5ee5e4"
server_key_version: "encryptionkey@0"
I20260812 14:55:52.022610 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:52.038394 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:52.038630 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:52.038753 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:52.038937 8118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:52.039222 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:52.039256 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.039283 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:52.039306 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.044358 8118 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.2:39165
I20260812 14:55:52.044435 8341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.2:39165 every 8 connection(s)
I20260812 14:55:52.044677 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/data/info.pb
I20260812 14:55:52.049326 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.045320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.2:51183 (local address 127.4.27.62:41111)
0812 14:55:52.045367 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.045369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.045559 (+ 190us) server_negotiation.cc:407] Connection header received
0812 14:55:52.046024 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.046025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.046038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.046084 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:52.046749 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.047238 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.048153 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.048352 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.048408 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.049057 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:52.049059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:52.049059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:52.049064 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:52.049078 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:52.049082 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:52.049127 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.049221 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.049247 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:55:52.049487 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.045299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41111 (local address 127.4.27.2:51183)
0812 14:55:52.045452 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.045481 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.045926 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.046167 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.046171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.046182 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:52.046659 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.046663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.047324 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.047325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.048066 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.048068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.048424 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.048425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.048547 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.048934 (+ 387us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:52.048934 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:52.048960 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:52.048980 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:52.049148 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:52.049153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:52.049156 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.049213 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.049307 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260812 14:55:52.049901 8344 heartbeater.cc:347] Connected to a master server at 127.4.27.62:41111
I20260812 14:55:52.049980 8344 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:52.050150 8344 heartbeater.cc:511] Master 127.4.27.62:41111 requested a full tablet report, sending...
I20260812 14:55:52.050482 7782 ts_manager.cc:194] Registered new tserver with Master: 5b275ee4c8ec46f68cfa47c293848405 (127.4.27.2:39165)
I20260812 14:55:52.050807 7782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.2:51183
I20260812 14:55:52.053455 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8118
I20260812 14:55:52.053550 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal/instance
I20260812 14:55:52.053728 4204 external_mini_cluster.cc:1428] Setting key 948d170d418fbda688e19cdc162b94ef
I20260812 14:55:52.054538 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.27.3:0
--local_ip_for_outbound_sockets=127.4.27.3
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41111
--builtin_ntp_servers=127.4.27.20:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 14:55:52.134948 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:52.135141 8347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:52.135169 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:52.136265 8347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:52.136309 8347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:52.136345 8347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.27.3
I20260812 14:55:52.137655 8347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.27.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.27.3
--webserver_port=0
--tserver_master_addrs=127.4.27.62:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.27.3
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:52.137885 8347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:52.138149 8347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:52.142786 8358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:52.142800 8347 server_base.cc:1034] running on GCE node
W20260812 14:55:52.142763 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:52.142753 8355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:52.143173 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:52.143394 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:52.144529 8347 hybrid_clock.cc:648] HybridClock initialized: now 1786546552144516 us; error 27 us; skew 500 ppm
I20260812 14:55:52.145689 8347 webserver.cc:460] Webserver started at http://127.4.27.3:34029/ using document root <none> and password file <none>
I20260812 14:55:52.145910 8347 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:52.145957 8347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:52.146085 8347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:52.147127 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data/instance:
uuid: "fe4c98d05cb34e5182308582c546045e"
format_stamp: "Formatted at 2026-08-12 14:55:52 on dist-test-slave-6zbq"
server_key: "b027450be861aae2bd4bdfcd6b9a20f5"
server_key_iv: "3ee9e8812199375e983876bee765e353"
server_key_version: "encryptionkey@0"
I20260812 14:55:52.147454 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance:
uuid: "fe4c98d05cb34e5182308582c546045e"
format_stamp: "Formatted at 2026-08-12 14:55:52 on dist-test-slave-6zbq"
server_key: "b027450be861aae2bd4bdfcd6b9a20f5"
server_key_iv: "3ee9e8812199375e983876bee765e353"
server_key_version: "encryptionkey@0"
I20260812 14:55:52.148620 8347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 14:55:52.149241 8368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.149405 8347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:52.149487 8347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal
uuid: "fe4c98d05cb34e5182308582c546045e"
format_stamp: "Formatted at 2026-08-12 14:55:52 on dist-test-slave-6zbq"
server_key: "b027450be861aae2bd4bdfcd6b9a20f5"
server_key_iv: "3ee9e8812199375e983876bee765e353"
server_key_version: "encryptionkey@0"
I20260812 14:55:52.149555 8347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:52.160717 8347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:52.160936 8347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:52.161046 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:52.161233 8347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 14:55:52.161514 8347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 14:55:52.161546 8347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.161571 8347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 14:55:52.161597 8347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:52.166936 8347 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.3:34643
I20260812 14:55:52.166989 8578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.3:34643 every 8 connection(s)
I20260812 14:55:52.167254 8347 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/data/info.pb
I20260812 14:55:52.169443 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8347
I20260812 14:55:52.169546 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal/instance
I20260812 14:55:52.169713 4204 external_mini_cluster.cc:1428] Setting key 9a0d6f21c24b80c89761f5e741b00adf
I20260812 14:55:52.172124 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.167854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:51683 (local address 127.4.27.62:41111)
0812 14:55:52.167902 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.167905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.168102 (+ 197us) server_negotiation.cc:407] Connection header received
0812 14:55:52.168531 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.168532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.168542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.168579 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:52.169142 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.169830 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.170974 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.171180 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.171234 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.171939 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:52.171940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:52.171940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:52.171944 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:52.171960 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:52.171964 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:52.172000 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.172044 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.172068 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:55:52.172333 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.167840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41111 (local address 127.4.27.3:51683)
0812 14:55:52.167998 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.168028 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.168447 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.168627 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.168629 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.168637 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:55:52.169083 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.169087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.169919 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.169920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.170899 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.170902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.171298 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.171299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.171413 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.171819 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 14:55:52.171819 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:55:52.171845 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 14:55:52.171867 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:55:52.172001 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:55:52.172003 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:55:52.172005 (+ 2us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.172045 (+ 40us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.172115 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260812 14:55:52.172744 8579 heartbeater.cc:347] Connected to a master server at 127.4.27.62:41111
I20260812 14:55:52.172824 8579 heartbeater.cc:464] Registering TS with master...
I20260812 14:55:52.172989 8579 heartbeater.cc:511] Master 127.4.27.62:41111 requested a full tablet report, sending...
I20260812 14:55:52.173302 7782 ts_manager.cc:194] Registered new tserver with Master: fe4c98d05cb34e5182308582c546045e (127.4.27.3:34643)
I20260812 14:55:52.173651 7782 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.27.3:51683
I20260812 14:55:52.181288 4204 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 14:55:52.185107 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.182549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.4.27.62:41111)
0812 14:55:52.182604 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.182606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.182692 (+ 86us) server_negotiation.cc:407] Connection header received
0812 14:55:52.182753 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.182754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.182766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.182810 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:52.182956 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.183414 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.184133 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.184331 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.184384 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.184894 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:52.184895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:52.184895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:52.184899 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:52.184912 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:52.184915 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:52.184960 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.185015 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.185037 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 14:55:52.186504 7782 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 14:55:52.187989 7782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 14:55:52.193926 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.189793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:41993 (local address 127.0.0.1:58010)
0812 14:55:52.190157 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.190166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.190216 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.190408 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.190410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.190413 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:52.190565 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.190568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.191877 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.191879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.192752 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.192755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.193635 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.193636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.193717 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.193731 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.193810 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.193859 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260812 14:55:52.193943 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.189962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.4.27.1:41993)
0812 14:55:52.190118 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.190122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.190223 (+ 101us) server_negotiation.cc:407] Connection header received
0812 14:55:52.190273 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.190275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.190291 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.190406 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:52.190654 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.191798 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.192824 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.193552 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.193632 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.193726 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.193824 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.193864 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260812 14:55:52.193957 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.190000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.4.27.3:34643)
0812 14:55:52.190154 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.190158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.190396 (+ 238us) server_negotiation.cc:407] Connection header received
0812 14:55:52.190537 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.190539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.190569 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.190633 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:52.190844 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.191487 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.192958 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.193591 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.193658 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.193725 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.193855 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.193893 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260812 14:55:52.194136 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.189817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.3:34643 (local address 127.0.0.1:47198)
0812 14:55:52.190312 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.190323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.190425 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.190694 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.190695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.190697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:52.190746 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.190747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.191596 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.191597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.192868 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.192872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.193668 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.193669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.193719 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.193731 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.194024 (+ 293us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.194069 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":82,"threads_started":1}
I20260812 14:55:52.194826 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.189793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:39165 (local address 127.0.0.1:34250)
0812 14:55:52.189945 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.189979 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.190041 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.190535 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.190536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.190538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:52.190569 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.190570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.191813 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.191814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.192873 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.192875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.194608 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.194609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.194668 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.194672 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.194721 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.194760 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260812 14:55:52.194993 8461 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd86f399459411ba7d9962e9d4b1cdf (DEFAULT_TABLE table=test-table [id=53c251c8ad574fa6be59391b724b1ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:52.195302 8461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd86f399459411ba7d9962e9d4b1cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:52.195202 7995 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd86f399459411ba7d9962e9d4b1cdf (DEFAULT_TABLE table=test-table [id=53c251c8ad574fa6be59391b724b1ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:52.195503 7995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd86f399459411ba7d9962e9d4b1cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:52.197688 8601 tablet_bootstrap.cc:492] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Bootstrap starting.
I20260812 14:55:52.197945 8602 tablet_bootstrap.cc:492] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: Bootstrap starting.
I20260812 14:55:52.198313 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.190138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34250 (local address 127.4.27.2:39165)
0812 14:55:52.190288 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.190291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.190300 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:55:52.190335 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.190357 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.190372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.190447 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:52.190656 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.191740 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.193763 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.194431 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.194488 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.194574 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.198186 (+ 3612us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.198237 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260812 14:55:52.198518 8601 tablet_bootstrap.cc:654] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:52.198570 8602 tablet_bootstrap.cc:654] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:52.198833 8602 log.cc:824] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:52.198836 8601 log.cc:824] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:52.198931 8223 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd86f399459411ba7d9962e9d4b1cdf (DEFAULT_TABLE table=test-table [id=53c251c8ad574fa6be59391b724b1ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 14:55:52.199271 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd86f399459411ba7d9962e9d4b1cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:52.199524 8601 tablet_bootstrap.cc:492] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: No bootstrap required, opened a new log
I20260812 14:55:52.199581 8601 ts_tablet_manager.cc:1397] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:52.201512 8601 raft_consensus.cc:348] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.201640 8601 raft_consensus.cc:374] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:52.201673 8601 raft_consensus.cc:729] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4c98d05cb34e5182308582c546045e, State: Initialized, Role: FOLLOWER
I20260812 14:55:52.201723 8607 tablet_bootstrap.cc:492] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Bootstrap starting.
I20260812 14:55:52.201782 8601 consensus_queue.cc:260] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.202035 8601 ts_tablet_manager.cc:1428] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 14:55:52.202090 8579 heartbeater.cc:503] Master 127.4.27.62:41111 was elected leader, sending a full tablet report...
I20260812 14:55:52.202292 8607 tablet_bootstrap.cc:654] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:52.202464 8602 tablet_bootstrap.cc:492] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: No bootstrap required, opened a new log
I20260812 14:55:52.202536 8602 ts_tablet_manager.cc:1397] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260812 14:55:52.202612 8607 log.cc:824] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:52.203295 8607 tablet_bootstrap.cc:492] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: No bootstrap required, opened a new log
I20260812 14:55:52.203361 8607 ts_tablet_manager.cc:1397] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 14:55:52.204610 8602 raft_consensus.cc:348] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.204727 8602 raft_consensus.cc:374] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:52.204749 8602 raft_consensus.cc:729] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b943f3100544ac972585776a6cde31, State: Initialized, Role: FOLLOWER
I20260812 14:55:52.204839 8602 consensus_queue.cc:260] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.205056 8602 ts_tablet_manager.cc:1428] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260812 14:55:52.205104 8113 heartbeater.cc:503] Master 127.4.27.62:41111 was elected leader, sending a full tablet report...
I20260812 14:55:52.205546 8607 raft_consensus.cc:348] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.205655 8607 raft_consensus.cc:374] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:52.205691 8607 raft_consensus.cc:729] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b275ee4c8ec46f68cfa47c293848405, State: Initialized, Role: FOLLOWER
I20260812 14:55:52.205827 8607 consensus_queue.cc:260] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.206079 8607 ts_tablet_manager.cc:1428] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260812 14:55:52.206130 8344 heartbeater.cc:503] Master 127.4.27.62:41111 was elected leader, sending a full tablet report...
W20260812 14:55:52.295156 8345 tablet.cc:2369] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:55:52.417757 8582 tablet.cc:2369] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 14:55:52.421118 8114 tablet.cc:2369] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:55:52.424724 8608 raft_consensus.cc:482] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 14:55:52.424813 8608 raft_consensus.cc:504] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.425105 8608 leader_election.cc:288] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43b943f3100544ac972585776a6cde31 (127.4.27.1:41993), 5b275ee4c8ec46f68cfa47c293848405 (127.4.27.2:39165)
I20260812 14:55:52.427874 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.425262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:41993 (local address 127.4.27.3:45967)
0812 14:55:52.425305 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.425317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.425356 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.425519 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.425520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.425522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:52.425592 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.425594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.426298 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.426299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.427103 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.427106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.427652 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.427653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.427727 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.427732 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.427783 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.427822 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":25}
I20260812 14:55:52.427888 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.425326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:45967 (local address 127.4.27.1:41993)
0812 14:55:52.425386 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.425388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.425398 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:55:52.425425 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.425426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.425436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.425483 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:52.425675 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.426231 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.427192 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.427586 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.427637 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.427646 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.427793 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.427825 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 14:55:52.428004 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.425271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.2:39165 (local address 127.4.27.3:50919)
0812 14:55:52.425566 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.425576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:55:52.425634 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:55:52.425776 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:55:52.425776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:55:52.425778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 14:55:52.425845 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.425848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.426482 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.426483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.427248 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:55:52.427251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.427770 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.427771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:55:52.427843 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.427847 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 14:55:52.427900 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 14:55:52.427934 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":130,"thread_start_us":92,"threads_started":1}
I20260812 14:55:52.428014 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.425334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.27.3:50919 (local address 127.4.27.2:39165)
0812 14:55:52.425397 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.425399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.425644 (+ 245us) server_negotiation.cc:407] Connection header received
0812 14:55:52.425702 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.425703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.425720 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.425761 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 14:55:52.425901 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.426394 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.427326 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.427683 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.427760 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.427770 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.427914 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.427950 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 14:55:52.428331 8031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd86f399459411ba7d9962e9d4b1cdf" candidate_uuid: "fe4c98d05cb34e5182308582c546045e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b943f3100544ac972585776a6cde31" is_pre_election: true
I20260812 14:55:52.428354 8249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd86f399459411ba7d9962e9d4b1cdf" candidate_uuid: "fe4c98d05cb34e5182308582c546045e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b275ee4c8ec46f68cfa47c293848405" is_pre_election: true
I20260812 14:55:52.428453 8031 raft_consensus.cc:2454] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe4c98d05cb34e5182308582c546045e in term 0.
I20260812 14:55:52.428483 8249 raft_consensus.cc:2454] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe4c98d05cb34e5182308582c546045e in term 0.
I20260812 14:55:52.428682 8374 leader_election.cc:302] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b275ee4c8ec46f68cfa47c293848405, fe4c98d05cb34e5182308582c546045e; no voters:
I20260812 14:55:52.428834 8608 raft_consensus.cc:2781] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 14:55:52.428895 8608 raft_consensus.cc:482] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 14:55:52.428911 8608 raft_consensus.cc:3037] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:52.429571 8608 raft_consensus.cc:504] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.429700 8608 leader_election.cc:288] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [CANDIDATE]: Term 1 election: Requested vote from peers 43b943f3100544ac972585776a6cde31 (127.4.27.1:41993), 5b275ee4c8ec46f68cfa47c293848405 (127.4.27.2:39165)
I20260812 14:55:52.429836 8031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd86f399459411ba7d9962e9d4b1cdf" candidate_uuid: "fe4c98d05cb34e5182308582c546045e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43b943f3100544ac972585776a6cde31"
I20260812 14:55:52.429847 8249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd86f399459411ba7d9962e9d4b1cdf" candidate_uuid: "fe4c98d05cb34e5182308582c546045e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b275ee4c8ec46f68cfa47c293848405"
I20260812 14:55:52.429912 8031 raft_consensus.cc:3037] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:52.429911 8249 raft_consensus.cc:3037] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:52.430766 8031 raft_consensus.cc:2454] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe4c98d05cb34e5182308582c546045e in term 1.
I20260812 14:55:52.430766 8249 raft_consensus.cc:2454] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe4c98d05cb34e5182308582c546045e in term 1.
I20260812 14:55:52.430931 8374 leader_election.cc:302] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b275ee4c8ec46f68cfa47c293848405, fe4c98d05cb34e5182308582c546045e; no voters:
I20260812 14:55:52.431034 8608 raft_consensus.cc:2781] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:52.431167 8608 raft_consensus.cc:686] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 1 LEADER]: Becoming Leader. State: Replica: fe4c98d05cb34e5182308582c546045e, State: Running, Role: LEADER
I20260812 14:55:52.431257 8608 consensus_queue.cc:237] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } }
I20260812 14:55:52.432010 7782 catalog_manager.cc:5495] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e reported cstate change: term changed from 0 to 1, leader changed from <none> to fe4c98d05cb34e5182308582c546045e (127.4.27.3). New cstate: current_term: 1 leader_uuid: "fe4c98d05cb34e5182308582c546045e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe4c98d05cb34e5182308582c546045e" member_type: VOTER last_known_addr { host: "127.4.27.3" port: 34643 } health_report { overall_health: HEALTHY } } }
I20260812 14:55:52.476807 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.474490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.4.27.3:34643)
0812 14:55:52.474645 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:52.474647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:52.474659 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:55:52.474696 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:52.474697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:52.474713 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:52.474791 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:55:52.474945 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.475394 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.476112 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:52.476292 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:52.476346 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:52.476500 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:55:52.476593 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:55:52.476671 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:52.476696 (+ 25us) server_negotiation.cc:299] Negotiation successful
0812 14:55:52.476714 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 14:55:52.478456 8249 raft_consensus.cc:1261] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 1 FOLLOWER]: Refusing update from remote peer fe4c98d05cb34e5182308582c546045e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:52.478457 8031 raft_consensus.cc:1261] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 1 FOLLOWER]: Refusing update from remote peer fe4c98d05cb34e5182308582c546045e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:55:52.478767 8608 consensus_queue.cc:1035] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43b943f3100544ac972585776a6cde31" member_type: VOTER last_known_addr { host: "127.4.27.1" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:52.478879 8616 consensus_queue.cc:1035] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b275ee4c8ec46f68cfa47c293848405" member_type: VOTER last_known_addr { host: "127.4.27.2" port: 39165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:55:52.481945 8625 mvcc.cc:205] Tried to move back new op lower bound from 7317694678949347328 to 7317694678758862848. Current Snapshot: MvccSnapshot[applied={T|T < 7317694678949347328 or (T in {7317694678949347328})}]
I20260812 14:55:52.483834 7778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:52.483914 7778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 14:55:52.484609 7778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 429f080089ad4f1fb6f30cf63be9a732: Sending DeleteTablet for 3 replicas of tablet 9dd86f399459411ba7d9962e9d4b1cdf
I20260812 14:55:52.484937 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd86f399459411ba7d9962e9d4b1cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:52 UTC) from {username='slave'} at 127.0.0.1:34250
I20260812 14:55:52.484958 7995 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd86f399459411ba7d9962e9d4b1cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:52 UTC) from {username='slave'} at 127.0.0.1:58010
I20260812 14:55:52.485159 8638 tablet_replica.cc:330] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31: stopping tablet replica
I20260812 14:55:52.485265 8638 raft_consensus.cc:2229] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:52.485337 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7879
I20260812 14:55:52.485327 8461 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd86f399459411ba7d9962e9d4b1cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 14:55:52 UTC) from {username='slave'} at 127.0.0.1:47198
I20260812 14:55:52.485365 8638 raft_consensus.cc:2258] T 9dd86f399459411ba7d9962e9d4b1cdf P 43b943f3100544ac972585776a6cde31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:52.485512 8637 tablet_replica.cc:330] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: stopping tablet replica
I20260812 14:55:52.485591 8637 raft_consensus.cc:2229] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 14:55:52.485684 8637 raft_consensus.cc:2258] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:52.490638 8637 ts_tablet_manager.cc:1905] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:52.492311 8637 ts_tablet_manager.cc:1918] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:52.492383 8637 log.cc:1196] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-1/wal/wals/9dd86f399459411ba7d9962e9d4b1cdf
I20260812 14:55:52.492616 8637 ts_tablet_manager.cc:1939] T 9dd86f399459411ba7d9962e9d4b1cdf P 5b275ee4c8ec46f68cfa47c293848405: Deleting consensus metadata
I20260812 14:55:52.493053 7756 catalog_manager.cc:4841] TS 5b275ee4c8ec46f68cfa47c293848405 (127.4.27.2:39165): tablet 9dd86f399459411ba7d9962e9d4b1cdf (table test-table [id=53c251c8ad574fa6be59391b724b1ea9]) successfully deleted
I20260812 14:55:52.493214 8639 tablet_replica.cc:330] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: stopping tablet replica
I20260812 14:55:52.493312 8639 raft_consensus.cc:2229] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 1 LEADER]: Raft consensus shutting down.
I20260812 14:55:52.493424 8639 raft_consensus.cc:2258] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 14:55:52.493758 8639 ts_tablet_manager.cc:1905] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 14:55:52.495307 8639 ts_tablet_manager.cc:1918] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 14:55:52.495378 8639 log.cc:1196] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Deleting WAL directory at /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786546313808590-4204-0/minicluster-data/ts-2/wal/wals/9dd86f399459411ba7d9962e9d4b1cdf
I20260812 14:55:52.495623 8639 ts_tablet_manager.cc:1939] T 9dd86f399459411ba7d9962e9d4b1cdf P fe4c98d05cb34e5182308582c546045e: Deleting consensus metadata
I20260812 14:55:52.496012 7759 catalog_manager.cc:4841] TS fe4c98d05cb34e5182308582c546045e (127.4.27.3:34643): tablet 9dd86f399459411ba7d9962e9d4b1cdf (table test-table [id=53c251c8ad574fa6be59391b724b1ea9]) successfully deleted
W20260812 14:55:52.506754 7758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.27.1:41993 (error 108)
I20260812 14:55:52.506981 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8118
I20260812 14:55:52.511168 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:52.511040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.1:41993 (local address 127.0.0.1:58016)
0812 14:55:52.511096 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0812 14:55:52.511135 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.1:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:55:52.511284 7758 catalog_manager.cc:4584] TS 43b943f3100544ac972585776a6cde31 (127.4.27.1:41993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dd86f399459411ba7d9962e9d4b1cdf: Network error: Client connection negotiation failed: client connection to 127.4.27.1:41993: connect: Connection refused (error 111)
I20260812 14:55:52.519080 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8347
I20260812 14:55:52.527689 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 7734
2026-08-12T14:55:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (889 ms)
[----------] 6 tests from EncryptionPolicyTest (9122 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:52 dist-test-slave-6zbq krb5kdc[8655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:52 dist-test-slave-6zbq krb5kdc[8655](info): set up 2 sockets
Aug 12 14:55:52 dist-test-slave-6zbq krb5kdc[8655](info): commencing operation
krb5kdc: starting...
W20260812 14:55:53.705241 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:53 dist-test-slave-6zbq krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:55:53.814895 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:53.893323 8672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:53.893503 8672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:53.893524 8672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:53.894577 8672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:53.894608 8672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:53.894620 8672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:53.894631 8672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:53.894641 8672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:53.896001 8672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:53.896173 8672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:53.896329 8672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:53.901072 8677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:53.901098 8680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:53.901078 8678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:53.901258 8672 server_base.cc:1034] running on GCE node
I20260812 14:55:53.901650 8672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:53.901896 8672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:53.903033 8672 hybrid_clock.cc:648] HybridClock initialized: now 1786546553903016 us; error 28 us; skew 500 ppm
Aug 12 14:55:53 dist-test-slave-6zbq krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:53.909839 8672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:55:53.910210 8672 webserver.cc:460] Webserver started at http://127.0.0.1:38249/ using document root <none> and password file <none>
I20260812 14:55:53.910439 8672 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:53.910490 8672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:53.910612 8672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:53.911592 8672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "9fd4dbe25b524bf1bb97aeefadf02695"
format_stamp: "Formatted at 2026-08-12 14:55:53 on dist-test-slave-6zbq"
server_key: "c6a2038f71f951e613519242baa2da3b"
server_key_iv: "51a88d1acaea649a0d3318023e9258bc"
server_key_version: "encryptionkey@0"
I20260812 14:55:53.911923 8672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "9fd4dbe25b524bf1bb97aeefadf02695"
format_stamp: "Formatted at 2026-08-12 14:55:53 on dist-test-slave-6zbq"
server_key: "c6a2038f71f951e613519242baa2da3b"
server_key_iv: "51a88d1acaea649a0d3318023e9258bc"
server_key_version: "encryptionkey@0"
I20260812 14:55:53.913100 8672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:55:53.913722 8690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:53.913870 8672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:55:53.913954 8672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "9fd4dbe25b524bf1bb97aeefadf02695"
format_stamp: "Formatted at 2026-08-12 14:55:53 on dist-test-slave-6zbq"
server_key: "c6a2038f71f951e613519242baa2da3b"
server_key_iv: "51a88d1acaea649a0d3318023e9258bc"
server_key_version: "encryptionkey@0"
I20260812 14:55:53.914012 8672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:53.924813 8672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:53.926759 8672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:53.926882 8672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:53.930466 8672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41877
I20260812 14:55:53.930510 8789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41877 every 8 connection(s)
I20260812 14:55:53.930806 8672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:53.931682 8790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:53.933799 8790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695: Bootstrap starting.
I20260812 14:55:53.934373 8790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:53.934661 8790 log.cc:824] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:53.935303 8790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695: No bootstrap required, opened a new log
I20260812 14:55:53.936849 8790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41877 } }
I20260812 14:55:53.936983 8790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:53.937004 8790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd4dbe25b524bf1bb97aeefadf02695, State: Initialized, Role: FOLLOWER
I20260812 14:55:53.937136 8790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41877 } }
I20260812 14:55:53.937202 8790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:53.937222 8790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:53.937251 8790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:53.937748 8790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41877 } }
I20260812 14:55:53.937844 8790 leader_election.cc:302] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fd4dbe25b524bf1bb97aeefadf02695; no voters:
I20260812 14:55:53.938000 8790 leader_election.cc:288] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:53.938038 8793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:53.938185 8793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [term 1 LEADER]: Becoming Leader. State: Replica: 9fd4dbe25b524bf1bb97aeefadf02695, State: Running, Role: LEADER
I20260812 14:55:53.938220 8790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:53.938272 8793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41877 } }
I20260812 14:55:53.938654 8797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fd4dbe25b524bf1bb97aeefadf02695. Latest consensus state: current_term: 1 leader_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41877 } } }
I20260812 14:55:53.938748 8797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:53.938868 8796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fd4dbe25b524bf1bb97aeefadf02695" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41877 } } }
I20260812 14:55:53.938954 8796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:53.939128 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8672
I20260812 14:55:53.939246 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:53.939442 4204 external_mini_cluster.cc:1428] Setting key ec8829a55bd37bcc397bb8689088f011
W20260812 14:55:53.939869 8823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:55:53.939924 8823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:55:53.939975 8809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:53.940146 8809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:53.941618 8809 catalog_manager.cc:1347] Generated new cluster ID: 49f5c2e5a9cc41f29f752d75904177d8
I20260812 14:55:53.941666 8809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:55:53 dist-test-slave-6zbq krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:55:53.950691 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:53.939917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35779 (local address 127.0.0.1:41877)
0812 14:55:53.940061 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:53.940070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:53.940248 (+ 178us) server_negotiation.cc:407] Connection header received
0812 14:55:53.940638 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:53.940646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:53.940702 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:53.940850 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:53.941576 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:53.942988 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:53.943995 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:53.944611 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:53.944720 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:53.947309 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:53.947313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:53.947314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:53.947325 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:53.949097 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:53.949512 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:53.949515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:53.949515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:53.949608 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:53.949818 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:53.949818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:53.949819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:53.950104 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:53.950237 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:53.950386 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 14:55:53.950477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260812 14:55:53.951540 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:55:53.958736 8809 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:53.959415 8809 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:53.959714 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:53.952908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47369 (local address 127.0.0.1:41877)
0812 14:55:53.952939 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:53.952941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:53.953038 (+ 97us) server_negotiation.cc:407] Connection header received
0812 14:55:53.953079 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:53.953079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:53.953088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:53.953142 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:53.953667 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:53.954444 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:53.955353 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:53.955670 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:53.955743 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:53.956884 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:53.956885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:53.956886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:53.956893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:53.958501 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:53.958908 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:53.958909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:53.958910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:53.958964 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:53.959163 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:53.959164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:53.959164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:53.959292 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:53.959377 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:53.959566 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:55:53.959632 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":16}
I20260812 14:55:53.963483 8809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fd4dbe25b524bf1bb97aeefadf02695: Generated new TSK 0
I20260812 14:55:53.963645 8809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:53.967535 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8672
W20260812 14:55:53.983371 8651 connection.cc:535] client connection to 127.0.0.1:41877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41877): Connection reset by peer (error 104)
2026-08-12T14:55:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:54 dist-test-slave-6zbq krb5kdc[8844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:54 dist-test-slave-6zbq krb5kdc[8844](info): set up 2 sockets
Aug 12 14:55:54 dist-test-slave-6zbq krb5kdc[8844](info): commencing operation
krb5kdc: starting...
W20260812 14:55:55.160775 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:55 dist-test-slave-6zbq krb5kdc[8844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:55:55.266974 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:55.346558 8859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:55.346751 8859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:55.346792 8859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:55.347884 8859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:55.347924 8859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:55.347942 8859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:55.347958 8859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:55.347975 8859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:55.349416 8859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:55.349647 8859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:55.349849 8859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:55.354513 8871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:55.354547 8874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:55.354547 8869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:55.354801 8859 server_base.cc:1034] running on GCE node
I20260812 14:55:55.354987 8859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:55.355226 8859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:55.356361 8859 hybrid_clock.cc:648] HybridClock initialized: now 1786546555356347 us; error 30 us; skew 500 ppm
Aug 12 14:55:55 dist-test-slave-6zbq krb5kdc[8844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:55.358796 8859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:55:55.359207 8859 webserver.cc:460] Webserver started at http://127.0.0.1:43271/ using document root <none> and password file <none>
I20260812 14:55:55.359429 8859 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:55.359478 8859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:55.359603 8859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:55.360596 8859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "97eea9693ead4b3987c7978ea0e7aef6"
format_stamp: "Formatted at 2026-08-12 14:55:55 on dist-test-slave-6zbq"
server_key: "e104aafd846fcd45a4987207c30fc86c"
server_key_iv: "d4cb9244eccfe5f0fb64bed4f650c459"
server_key_version: "encryptionkey@0"
I20260812 14:55:55.360965 8859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "97eea9693ead4b3987c7978ea0e7aef6"
format_stamp: "Formatted at 2026-08-12 14:55:55 on dist-test-slave-6zbq"
server_key: "e104aafd846fcd45a4987207c30fc86c"
server_key_iv: "d4cb9244eccfe5f0fb64bed4f650c459"
server_key_version: "encryptionkey@0"
I20260812 14:55:55.362201 8859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:55:55.362900 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:55.363054 8859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:55.363126 8859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "97eea9693ead4b3987c7978ea0e7aef6"
format_stamp: "Formatted at 2026-08-12 14:55:55 on dist-test-slave-6zbq"
server_key: "e104aafd846fcd45a4987207c30fc86c"
server_key_iv: "d4cb9244eccfe5f0fb64bed4f650c459"
server_key_version: "encryptionkey@0"
I20260812 14:55:55.363193 8859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:55.379065 8859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:55.380731 8859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:55.380861 8859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:55.384603 8859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41903
I20260812 14:55:55.384642 8987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41903 every 8 connection(s)
I20260812 14:55:55.384945 8859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:55.385885 8988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:55.388085 8988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6: Bootstrap starting.
I20260812 14:55:55.388595 8988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:55.388818 8988 log.cc:824] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:55.389429 8988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6: No bootstrap required, opened a new log
I20260812 14:55:55.391016 8988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eea9693ead4b3987c7978ea0e7aef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41903 } }
I20260812 14:55:55.391116 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 8859
I20260812 14:55:55.391158 8988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:55.391180 8988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97eea9693ead4b3987c7978ea0e7aef6, State: Initialized, Role: FOLLOWER
I20260812 14:55:55.391242 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:55.391304 8988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eea9693ead4b3987c7978ea0e7aef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41903 } }
I20260812 14:55:55.391373 8988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:55.391409 8988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:55.391438 4204 external_mini_cluster.cc:1428] Setting key cb2e80d7ae45e76f8eb2582de925e246
I20260812 14:55:55.391453 8988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:55.392071 8988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eea9693ead4b3987c7978ea0e7aef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41903 } }
I20260812 14:55:55.392170 8988 leader_election.cc:302] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97eea9693ead4b3987c7978ea0e7aef6; no voters:
I20260812 14:55:55.392338 8988 leader_election.cc:288] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:55.392366 8997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:55.392496 8997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [term 1 LEADER]: Becoming Leader. State: Replica: 97eea9693ead4b3987c7978ea0e7aef6, State: Running, Role: LEADER
I20260812 14:55:55.392581 8988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:55.392619 8997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eea9693ead4b3987c7978ea0e7aef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41903 } }
I20260812 14:55:55.392989 8997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97eea9693ead4b3987c7978ea0e7aef6. Latest consensus state: current_term: 1 leader_uuid: "97eea9693ead4b3987c7978ea0e7aef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eea9693ead4b3987c7978ea0e7aef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41903 } } }
I20260812 14:55:55.393201 8997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:55.393131 8998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97eea9693ead4b3987c7978ea0e7aef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97eea9693ead4b3987c7978ea0e7aef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41903 } } }
I20260812 14:55:55.393577 8998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:55.393741 9016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:55.394291 9016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:55.395637 9016 catalog_manager.cc:1347] Generated new cluster ID: b65a0f8a02b64cd7a9867a67bcfaf4ca
I20260812 14:55:55.395685 9016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:55.397691 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:55.392184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46695 (local address 127.0.0.1:41903)
0812 14:55:55.392352 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:55.392362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:55.392397 (+ 35us) server_negotiation.cc:407] Connection header received
0812 14:55:55.392515 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:55.392522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:55.392575 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:55.392966 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:55.393066 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:55.394626 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:55.395695 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:55.396266 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:55.396369 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:55.397055 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:55.397059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:55.397060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:55.397249 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:55.397286 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:55.397299 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:55.397371 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:55.397475 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 14:55:55.397499 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260812 14:55:55.398598 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:55:55.403546 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:55.399965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39553 (local address 127.0.0.1:41903)
0812 14:55:55.399994 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:55.399996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:55.400117 (+ 121us) server_negotiation.cc:407] Connection header received
0812 14:55:55.400164 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:55.400164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:55.400172 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:55.400225 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:55.400346 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:55.401343 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:55.402322 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:55.402644 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:55.402697 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:55.403315 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:55.403316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:55.403317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:55.403321 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:55.403333 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:55.403338 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:55.403388 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:55.403470 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 14:55:55.403481 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 14:55:55.417598 9016 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:55.418081 9016 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:55.423139 9016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97eea9693ead4b3987c7978ea0e7aef6: Generated new TSK 0
I20260812 14:55:55.423292 9016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:55.425565 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 8859
W20260812 14:55:55.442587 8838 connection.cc:535] client connection to 127.0.0.1:41903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41903): Connection reset by peer (error 104)
2026-08-12T14:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:55 dist-test-slave-6zbq krb5kdc[9050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:55 dist-test-slave-6zbq krb5kdc[9050](info): set up 2 sockets
Aug 12 14:55:55 dist-test-slave-6zbq krb5kdc[9050](info): commencing operation
krb5kdc: starting...
W20260812 14:55:56.617157 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:56 dist-test-slave-6zbq krb5kdc[9050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:55:56.727980 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:56.806109 9070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:56.806294 9070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:56.806315 9070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:56.807451 9070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:56.807492 9070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:56.807504 9070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:56.807524 9070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:56.807548 9070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:56.808948 9070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:56.809152 9070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:56.809345 9070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:56.813983 9081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:56.814007 9078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:56.814157 9070 server_base.cc:1034] running on GCE node
W20260812 14:55:56.814083 9087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:56.814456 9070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:56.814725 9070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:56.815887 9070 hybrid_clock.cc:648] HybridClock initialized: now 1786546556815874 us; error 29 us; skew 500 ppm
Aug 12 14:55:56 dist-test-slave-6zbq krb5kdc[9050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:56.818223 9070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:55:56.818668 9070 webserver.cc:460] Webserver started at http://127.0.0.1:36013/ using document root <none> and password file <none>
I20260812 14:55:56.818868 9070 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:56.818910 9070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:56.819029 9070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:56.820017 9070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "b1874493e33a4409b125ff51ee09e59d"
format_stamp: "Formatted at 2026-08-12 14:55:56 on dist-test-slave-6zbq"
server_key: "5e6feb1b460863b5102b9f0d66013c09"
server_key_iv: "d2a1318dbad3da32592157e53f13f0d7"
server_key_version: "encryptionkey@0"
I20260812 14:55:56.820328 9070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "b1874493e33a4409b125ff51ee09e59d"
format_stamp: "Formatted at 2026-08-12 14:55:56 on dist-test-slave-6zbq"
server_key: "5e6feb1b460863b5102b9f0d66013c09"
server_key_iv: "d2a1318dbad3da32592157e53f13f0d7"
server_key_version: "encryptionkey@0"
I20260812 14:55:56.821508 9070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:55:56.822223 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:56.822386 9070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:56.822463 9070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "b1874493e33a4409b125ff51ee09e59d"
format_stamp: "Formatted at 2026-08-12 14:55:56 on dist-test-slave-6zbq"
server_key: "5e6feb1b460863b5102b9f0d66013c09"
server_key_iv: "d2a1318dbad3da32592157e53f13f0d7"
server_key_version: "encryptionkey@0"
I20260812 14:55:56.822527 9070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:56.825544 9070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:56.827308 9070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:56.827417 9070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:56.831080 9070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35507
I20260812 14:55:56.831110 9193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35507 every 8 connection(s)
I20260812 14:55:56.831374 9070 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:56.831779 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9070
I20260812 14:55:56.831923 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:56.832118 4204 external_mini_cluster.cc:1428] Setting key 7445c1316c22499f3a01b5274c2b1623
I20260812 14:55:56.832438 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:56.834533 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:56.832584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59371 (local address 127.0.0.1:35507)
0812 14:55:56.832760 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:56.832769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:56.832800 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:55:56.833455 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:56.833463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:56.833497 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:56.833650 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:56.834120 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:56.834123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:56.834123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:56.834127 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:56.834166 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:56.834181 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:56.834269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:56.834304 (+ 35us) server_negotiation.cc:299] Negotiation successful
0812 14:55:56.834361 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260812 14:55:56.835477 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:55:56.835608 9194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d: Bootstrap starting.
I20260812 14:55:56.836262 9194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:56.836573 9194 log.cc:824] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:56.837248 9194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d: No bootstrap required, opened a new log
I20260812 14:55:56.837845 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:56.836950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58867 (local address 127.0.0.1:35507)
0812 14:55:56.836979 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:56.836981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:56.837092 (+ 111us) server_negotiation.cc:407] Connection header received
0812 14:55:56.837152 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:56.837153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:56.837157 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:56.837204 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:56.837664 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:56.837665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:56.837665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:56.837669 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:56.837681 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:56.837685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:56.837730 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:56.837779 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 14:55:56.837791 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260812 14:55:56.839247 9194 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1874493e33a4409b125ff51ee09e59d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35507 } }
I20260812 14:55:56.839373 9194 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:56.839414 9194 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1874493e33a4409b125ff51ee09e59d, State: Initialized, Role: FOLLOWER
I20260812 14:55:56.839512 9194 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1874493e33a4409b125ff51ee09e59d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35507 } }
I20260812 14:55:56.839574 9194 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:56.839598 9194 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:56.839627 9194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:56.840073 9194 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1874493e33a4409b125ff51ee09e59d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35507 } }
I20260812 14:55:56.840154 9194 leader_election.cc:302] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1874493e33a4409b125ff51ee09e59d; no voters:
I20260812 14:55:56.840281 9194 leader_election.cc:288] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:56.840341 9214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:56.840490 9194 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:56.840502 9214 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [term 1 LEADER]: Becoming Leader. State: Replica: b1874493e33a4409b125ff51ee09e59d, State: Running, Role: LEADER
I20260812 14:55:56.840703 9214 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1874493e33a4409b125ff51ee09e59d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35507 } }
I20260812 14:55:56.841073 9215 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1874493e33a4409b125ff51ee09e59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1874493e33a4409b125ff51ee09e59d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35507 } } }
I20260812 14:55:56.841180 9215 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:56.841337 9216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1874493e33a4409b125ff51ee09e59d. Latest consensus state: current_term: 1 leader_uuid: "b1874493e33a4409b125ff51ee09e59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1874493e33a4409b125ff51ee09e59d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35507 } } }
I20260812 14:55:56.841382 9223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:56.841400 9216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:56.841951 9223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:56.843717 9223 catalog_manager.cc:1347] Generated new cluster ID: 59a71c07ee2846028ae6a47b771e86c0
I20260812 14:55:56.843823 9223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:56.849941 9223 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:56.850103 9223 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:56.862207 9223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1874493e33a4409b125ff51ee09e59d: Generated new TSK 0
I20260812 14:55:56.862453 9223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:56.870970 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9070
2026-08-12T14:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1444 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:56 dist-test-slave-6zbq krb5kdc[9240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:56 dist-test-slave-6zbq krb5kdc[9240](info): set up 2 sockets
Aug 12 14:55:56 dist-test-slave-6zbq krb5kdc[9240](info): commencing operation
krb5kdc: starting...
W20260812 14:55:58.051699 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:58 dist-test-slave-6zbq krb5kdc[9240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:55:58.158501 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:58.237182 9258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:58.237363 9258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:58.237385 9258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:58.238502 9258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:58.238543 9258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:58.238556 9258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:58.238579 9258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:58.238602 9258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:58.240041 9258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:58.240255 9258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:58.240474 9258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:58.244921 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:58.244935 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:58.244949 9258 server_base.cc:1034] running on GCE node
W20260812 14:55:58.244935 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:58.245301 9258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:58.245565 9258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:58.246699 9258 hybrid_clock.cc:648] HybridClock initialized: now 1786546558246691 us; error 29 us; skew 500 ppm
Aug 12 14:55:58 dist-test-slave-6zbq krb5kdc[9240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:58.249006 9258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:55:58.249372 9258 webserver.cc:460] Webserver started at http://127.0.0.1:45535/ using document root <none> and password file <none>
I20260812 14:55:58.249557 9258 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:58.249595 9258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:58.249682 9258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:58.250686 9258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "dbb7edeefee940df9e4f06f14d169aa0"
format_stamp: "Formatted at 2026-08-12 14:55:58 on dist-test-slave-6zbq"
server_key: "f867ef08b44459c58d34b086c512b3bd"
server_key_iv: "b259db980e19d6da0cc65bd38b64c405"
server_key_version: "encryptionkey@0"
I20260812 14:55:58.250980 9258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "dbb7edeefee940df9e4f06f14d169aa0"
format_stamp: "Formatted at 2026-08-12 14:55:58 on dist-test-slave-6zbq"
server_key: "f867ef08b44459c58d34b086c512b3bd"
server_key_iv: "b259db980e19d6da0cc65bd38b64c405"
server_key_version: "encryptionkey@0"
I20260812 14:55:58.252188 9258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:55:58.252805 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:58.252964 9258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:58.253036 9258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "dbb7edeefee940df9e4f06f14d169aa0"
format_stamp: "Formatted at 2026-08-12 14:55:58 on dist-test-slave-6zbq"
server_key: "f867ef08b44459c58d34b086c512b3bd"
server_key_iv: "b259db980e19d6da0cc65bd38b64c405"
server_key_version: "encryptionkey@0"
I20260812 14:55:58.253095 9258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:58.269084 9258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:58.270977 9258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:58.271099 9258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:58.274634 9258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42917
I20260812 14:55:58.274667 9396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42917 every 8 connection(s)
I20260812 14:55:58.274955 9258 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:58.275838 9397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:58.277961 9397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0: Bootstrap starting.
I20260812 14:55:58.278537 9397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:58.278795 9397 log.cc:824] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:58.279500 9397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0: No bootstrap required, opened a new log
I20260812 14:55:58.281023 9397 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb7edeefee940df9e4f06f14d169aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260812 14:55:58.281172 9397 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:58.281208 9397 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb7edeefee940df9e4f06f14d169aa0, State: Initialized, Role: FOLLOWER
I20260812 14:55:58.281342 9397 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb7edeefee940df9e4f06f14d169aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260812 14:55:58.281418 9397 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:58.281453 9397 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:58.281500 9397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:58.281981 9397 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb7edeefee940df9e4f06f14d169aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260812 14:55:58.282069 9397 leader_election.cc:302] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbb7edeefee940df9e4f06f14d169aa0; no voters:
I20260812 14:55:58.282239 9397 leader_election.cc:288] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:58.282271 9400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:58.282480 9397 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:58.282513 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9258
I20260812 14:55:58.282629 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:58.282783 4204 external_mini_cluster.cc:1428] Setting key d24dc5229e6e73efa71e9aacef389997
I20260812 14:55:58.283025 9400 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [term 1 LEADER]: Becoming Leader. State: Replica: dbb7edeefee940df9e4f06f14d169aa0, State: Running, Role: LEADER
I20260812 14:55:58.283178 9400 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb7edeefee940df9e4f06f14d169aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } }
I20260812 14:55:58.283584 9411 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb7edeefee940df9e4f06f14d169aa0. Latest consensus state: current_term: 1 leader_uuid: "dbb7edeefee940df9e4f06f14d169aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb7edeefee940df9e4f06f14d169aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } } }
I20260812 14:55:58.283667 9411 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:58.283962 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbb7edeefee940df9e4f06f14d169aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb7edeefee940df9e4f06f14d169aa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42917 } } }
I20260812 14:55:58.284415 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0 [sys.catalog]: This master's current role is: LEADER
W20260812 14:55:58.284417 9425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:55:58.284495 9425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:55:58.284569 9424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:58.284742 9424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:58.286155 9424 catalog_manager.cc:1347] Generated new cluster ID: 1edc129de9a94a4e9cff7f13953a9971
I20260812 14:55:58.286240 9424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:55:58 dist-test-slave-6zbq krb5kdc[9240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:55:58.293728 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:58.283425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42661 (local address 127.0.0.1:42917)
0812 14:55:58.283609 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:58.283619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:58.283679 (+ 60us) server_negotiation.cc:407] Connection header received
0812 14:55:58.283842 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:58.283849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:58.283907 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:58.284035 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:58.284899 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:58.286002 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:58.287053 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:58.287827 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:58.287936 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:58.290470 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:58.290475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:58.290475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:58.290486 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:58.292245 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:58.292619 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:58.292621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:58.292621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:58.292729 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:58.292910 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:58.292910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:58.292911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:58.293166 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:58.293309 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:58.293404 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:55:58.293524 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260812 14:55:58.294538 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:55:58.296217 9424 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:58.296689 9424 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:58.301774 9424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb7edeefee940df9e4f06f14d169aa0: Generated new TSK 0
I20260812 14:55:58.301927 9424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:58.302636 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:58.295961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37713 (local address 127.0.0.1:42917)
0812 14:55:58.295998 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:58.296000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:58.296041 (+ 41us) server_negotiation.cc:407] Connection header received
0812 14:55:58.296095 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:58.296096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:58.296116 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:58.296163 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:58.296864 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:58.297552 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:58.298543 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:58.298841 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:58.298894 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:58.300253 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:58.300255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:58.300255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:55:58.300261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:58.301638 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:58.301968 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:58.301969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:58.301970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:58.302022 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:55:58.302182 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:55:58.302183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:55:58.302183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:55:58.302282 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:58.302381 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:58.302483 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 14:55:58.302563 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:55:58.304407 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9258
2026-08-12T14:55:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:58 dist-test-slave-6zbq krb5kdc[9446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:58 dist-test-slave-6zbq krb5kdc[9446](info): set up 2 sockets
Aug 12 14:55:58 dist-test-slave-6zbq krb5kdc[9446](info): commencing operation
krb5kdc: starting...
W20260812 14:55:59.489178 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:55:59 dist-test-slave-6zbq krb5kdc[9446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:55:59.596742 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:55:59.674738 9468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:55:59.674923 9468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:55:59.674954 9468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:55:59.676039 9468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:55:59.676079 9468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:55:59.676097 9468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:55:59.676115 9468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:55:59.676131 9468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:55:59.677567 9468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:55:59.677788 9468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:55:59.677974 9468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:55:59.682502 9477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:59.682492 9476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:55:59.682490 9480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:55:59.682552 9468 server_base.cc:1034] running on GCE node
I20260812 14:55:59.682889 9468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:55:59.683146 9468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:55:59.684275 9468 hybrid_clock.cc:648] HybridClock initialized: now 1786546559684261 us; error 32 us; skew 500 ppm
Aug 12 14:55:59 dist-test-slave-6zbq krb5kdc[9446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:55:59.691545 9468 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:55:59.691931 9468 webserver.cc:460] Webserver started at http://127.0.0.1:36729/ using document root <none> and password file <none>
I20260812 14:55:59.692114 9468 fs_manager.cc:359] Metadata directory not provided
I20260812 14:55:59.692162 9468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:55:59.692278 9468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:55:59.693226 9468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "23db53a57ed240d998960c86d5f34e7f"
format_stamp: "Formatted at 2026-08-12 14:55:59 on dist-test-slave-6zbq"
server_key: "d3d2a7ba11b7fd9ab1217776da4d9e58"
server_key_iv: "1ec1add3aaea1e13c33e5af60525fc6e"
server_key_version: "encryptionkey@0"
I20260812 14:55:59.693538 9468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "23db53a57ed240d998960c86d5f34e7f"
format_stamp: "Formatted at 2026-08-12 14:55:59 on dist-test-slave-6zbq"
server_key: "d3d2a7ba11b7fd9ab1217776da4d9e58"
server_key_iv: "1ec1add3aaea1e13c33e5af60525fc6e"
server_key_version: "encryptionkey@0"
I20260812 14:55:59.694725 9468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:55:59.695333 9488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:55:59.695477 9468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:55:59.695559 9468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "23db53a57ed240d998960c86d5f34e7f"
format_stamp: "Formatted at 2026-08-12 14:55:59 on dist-test-slave-6zbq"
server_key: "d3d2a7ba11b7fd9ab1217776da4d9e58"
server_key_iv: "1ec1add3aaea1e13c33e5af60525fc6e"
server_key_version: "encryptionkey@0"
I20260812 14:55:59.695614 9468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:55:59.701831 9468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:55:59.703460 9468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:55:59.703583 9468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:55:59.707190 9468 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34767
I20260812 14:55:59.707237 9591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34767 every 8 connection(s)
I20260812 14:55:59.707508 9468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:55:59.708355 9593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:55:59.710404 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f: Bootstrap starting.
I20260812 14:55:59.710709 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9468
I20260812 14:55:59.710834 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:55:59.710999 4204 external_mini_cluster.cc:1428] Setting key f9f88d903b9dd7b09b0b5d5cf067b472
I20260812 14:55:59.711081 9593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f: Neither blocks nor log segments found. Creating new log.
I20260812 14:55:59.711359 9593 log.cc:824] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f: Log is configured to *not* fsync() on all Append() calls
I20260812 14:55:59.712020 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f: No bootstrap required, opened a new log
I20260812 14:55:59.713944 9593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db53a57ed240d998960c86d5f34e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260812 14:55:59.714099 9593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:55:59.714128 9593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23db53a57ed240d998960c86d5f34e7f, State: Initialized, Role: FOLLOWER
I20260812 14:55:59.714236 9593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db53a57ed240d998960c86d5f34e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260812 14:55:59.714301 9593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:55:59.714334 9593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:55:59.714406 9593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:55:59.714905 9593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db53a57ed240d998960c86d5f34e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260812 14:55:59.715008 9593 leader_election.cc:302] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23db53a57ed240d998960c86d5f34e7f; no voters:
I20260812 14:55:59.715162 9593 leader_election.cc:288] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:55:59.715229 9601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:55:59.715554 9601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [term 1 LEADER]: Becoming Leader. State: Replica: 23db53a57ed240d998960c86d5f34e7f, State: Running, Role: LEADER
I20260812 14:55:59.715674 9593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:55:59.715673 9601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db53a57ed240d998960c86d5f34e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260812 14:55:59.716272 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:59.711488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59729 (local address 127.0.0.1:34767)
0812 14:55:59.711649 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:59.711658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:59.711688 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:55:59.712124 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:59.712133 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:59.712190 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:59.712296 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:59.712528 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:59.713361 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:59.714511 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:59.714892 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:59.714969 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:59.715611 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:59.715614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:59.715614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:59.715798 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:59.715835 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:59.715849 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:59.715904 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:59.716024 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 14:55:59.716087 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260812 14:55:59.716385 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23db53a57ed240d998960c86d5f34e7f. Latest consensus state: current_term: 1 leader_uuid: "23db53a57ed240d998960c86d5f34e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db53a57ed240d998960c86d5f34e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } } }
I20260812 14:55:59.716542 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:59.716713 9604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23db53a57ed240d998960c86d5f34e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23db53a57ed240d998960c86d5f34e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } } }
I20260812 14:55:59.716743 9616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:55:59.717275 9616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:55:59.717576 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:55:59.716830 9604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f [sys.catalog]: This master's current role is: LEADER
I20260812 14:55:59.719151 9616 catalog_manager.cc:1347] Generated new cluster ID: 200c5b9500c94af5a7fd88d944d59274
I20260812 14:55:59.719193 9616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:55:59.722493 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:55:59.718989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35815 (local address 127.0.0.1:34767)
0812 14:55:59.719021 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:55:59.719022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:55:59.719269 (+ 247us) server_negotiation.cc:407] Connection header received
0812 14:55:59.719363 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:55:59.719364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:55:59.719373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:55:59.719421 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:55:59.719546 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:59.720243 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:59.721274 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:55:59.721624 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:55:59.721675 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:55:59.722254 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:55:59.722255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:55:59.722255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:55:59.722259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:55:59.722272 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:55:59.722276 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:55:59.722319 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:55:59.722409 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 14:55:59.722429 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:55:59.726851 9616 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:55:59.727313 9616 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:55:59.732056 9616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23db53a57ed240d998960c86d5f34e7f: Generated new TSK 0
I20260812 14:55:59.732193 9616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:55:59.734941 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9468
2026-08-12T14:55:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:55:59 dist-test-slave-6zbq krb5kdc[9650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:55:59 dist-test-slave-6zbq krb5kdc[9650](info): set up 2 sockets
Aug 12 14:55:59 dist-test-slave-6zbq krb5kdc[9650](info): commencing operation
krb5kdc: starting...
W20260812 14:56:00.907727 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:00 dist-test-slave-6zbq krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:01.014067 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:01.091516 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:01.091698 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:01.091734 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:01.092810 9670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:01.092849 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:01.092866 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:01.092881 9670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:01.092898 9670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:01.094317 9670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:01.094573 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:01.094765 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:01.099251 9679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:01.099280 9683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:01.099331 9681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:01.099313 9670 server_base.cc:1034] running on GCE node
I20260812 14:56:01.099553 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:01.099817 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:01.100958 9670 hybrid_clock.cc:648] HybridClock initialized: now 1786546561100946 us; error 30 us; skew 500 ppm
Aug 12 14:56:01 dist-test-slave-6zbq krb5kdc[9650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:01.103299 9670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:01.103726 9670 webserver.cc:460] Webserver started at http://127.0.0.1:37009/ using document root <none> and password file <none>
I20260812 14:56:01.103914 9670 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:01.103961 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:01.104081 9670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:01.105065 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "c19e78d6ed5444a39992b1c50258bc41"
format_stamp: "Formatted at 2026-08-12 14:56:01 on dist-test-slave-6zbq"
server_key: "1f4881a37e2158cae5eb10a481152715"
server_key_iv: "d2b9e36cff88b56ef676a496a2bee4e6"
server_key_version: "encryptionkey@0"
I20260812 14:56:01.105414 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "c19e78d6ed5444a39992b1c50258bc41"
format_stamp: "Formatted at 2026-08-12 14:56:01 on dist-test-slave-6zbq"
server_key: "1f4881a37e2158cae5eb10a481152715"
server_key_iv: "d2b9e36cff88b56ef676a496a2bee4e6"
server_key_version: "encryptionkey@0"
I20260812 14:56:01.106611 9670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:56:01.107296 9691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:01.107456 9670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:01.107543 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "c19e78d6ed5444a39992b1c50258bc41"
format_stamp: "Formatted at 2026-08-12 14:56:01 on dist-test-slave-6zbq"
server_key: "1f4881a37e2158cae5eb10a481152715"
server_key_iv: "d2b9e36cff88b56ef676a496a2bee4e6"
server_key_version: "encryptionkey@0"
I20260812 14:56:01.107615 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:01.110710 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:01.112665 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:01.112789 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:01.116407 9670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45859
I20260812 14:56:01.116438 9773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45859 every 8 connection(s)
I20260812 14:56:01.116715 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:01.117604 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:01.117812 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9670
I20260812 14:56:01.117918 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:01.118063 4204 external_mini_cluster.cc:1428] Setting key 3562ab89540b72e0cfc13a8eab3f0d3f
I20260812 14:56:01.120298 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:01.118528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41495 (local address 127.0.0.1:45859)
0812 14:56:01.118694 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:01.118703 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:01.118737 (+ 34us) server_negotiation.cc:407] Connection header received
0812 14:56:01.119231 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:01.119240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:01.119274 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:01.119418 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:01.119917 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:01.119921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:01.119921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:01.119925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:01.119965 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:01.119979 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:01.120057 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:01.120089 (+ 32us) server_negotiation.cc:299] Negotiation successful
0812 14:56:01.120125 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260812 14:56:01.120455 9774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41: Bootstrap starting.
I20260812 14:56:01.121097 9774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:01.121160 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:01.121369 9774 log.cc:824] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:01.121966 9774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41: No bootstrap required, opened a new log
I20260812 14:56:01.123456 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:01.122484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55967 (local address 127.0.0.1:45859)
0812 14:56:01.122515 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:01.122517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:01.122664 (+ 147us) server_negotiation.cc:407] Connection header received
0812 14:56:01.122733 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:01.122733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:01.122738 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:01.122780 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:01.123266 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:01.123267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:01.123268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:01.123272 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:01.123285 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:01.123289 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:01.123334 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:01.123384 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:56:01.123397 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 14:56:01.124019 9774 raft_consensus.cc:348] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19e78d6ed5444a39992b1c50258bc41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 14:56:01.124161 9774 raft_consensus.cc:374] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:01.124184 9774 raft_consensus.cc:729] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19e78d6ed5444a39992b1c50258bc41, State: Initialized, Role: FOLLOWER
I20260812 14:56:01.124275 9774 consensus_queue.cc:260] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19e78d6ed5444a39992b1c50258bc41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 14:56:01.124359 9774 raft_consensus.cc:388] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:01.124384 9774 raft_consensus.cc:482] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:01.124415 9774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:01.124873 9774 raft_consensus.cc:504] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19e78d6ed5444a39992b1c50258bc41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 14:56:01.124956 9774 leader_election.cc:302] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c19e78d6ed5444a39992b1c50258bc41; no voters:
I20260812 14:56:01.125077 9774 leader_election.cc:288] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:01.125133 9802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:01.125288 9774 sys_catalog.cc:564] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:01.125324 9802 raft_consensus.cc:686] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [term 1 LEADER]: Becoming Leader. State: Replica: c19e78d6ed5444a39992b1c50258bc41, State: Running, Role: LEADER
I20260812 14:56:01.125445 9802 consensus_queue.cc:237] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19e78d6ed5444a39992b1c50258bc41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 14:56:01.125783 9805 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c19e78d6ed5444a39992b1c50258bc41. Latest consensus state: current_term: 1 leader_uuid: "c19e78d6ed5444a39992b1c50258bc41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19e78d6ed5444a39992b1c50258bc41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } } }
I20260812 14:56:01.125895 9805 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:01.125787 9804 sys_catalog.cc:455] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c19e78d6ed5444a39992b1c50258bc41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c19e78d6ed5444a39992b1c50258bc41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } } }
I20260812 14:56:01.126016 9804 sys_catalog.cc:458] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41 [sys.catalog]: This master's current role is: LEADER
W20260812 14:56:01.126976 9830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:01.127033 9830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:01.127081 9821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:01.127230 9821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:01.128595 9821 catalog_manager.cc:1347] Generated new cluster ID: f90a4cfc06ec40a8aa1c1e49841ca681
I20260812 14:56:01.128638 9821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:01.150240 9821 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:01.150434 9821 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:01.156821 9821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c19e78d6ed5444a39992b1c50258bc41: Generated new TSK 0
I20260812 14:56:01.156997 9821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:01.167397 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9670
2026-08-12T14:56:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:01 dist-test-slave-6zbq krb5kdc[9840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:01 dist-test-slave-6zbq krb5kdc[9840](info): set up 2 sockets
Aug 12 14:56:01 dist-test-slave-6zbq krb5kdc[9840](info): commencing operation
krb5kdc: starting...
W20260812 14:56:02.340412 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:02 dist-test-slave-6zbq krb5kdc[9840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:02.448068 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:02.526711 9865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:02.526903 9865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:02.526937 9865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:02.528067 9865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:02.528110 9865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:02.528127 9865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:02.528144 9865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:02.528159 9865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:02.529599 9865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:02.529843 9865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:02.530043 9865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:02.534780 9875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:02.534786 9881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:02.534816 9865 server_base.cc:1034] running on GCE node
W20260812 14:56:02.534806 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:02.535269 9865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:02.535547 9865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:02.536693 9865 hybrid_clock.cc:648] HybridClock initialized: now 1786546562536682 us; error 30 us; skew 500 ppm
Aug 12 14:56:02 dist-test-slave-6zbq krb5kdc[9840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:02.539000 9865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:02.539372 9865 webserver.cc:460] Webserver started at http://127.0.0.1:42051/ using document root <none> and password file <none>
I20260812 14:56:02.539582 9865 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:02.539629 9865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:02.539752 9865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:02.540711 9865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "4873c2c4873c41498092ccc68ce5476e"
format_stamp: "Formatted at 2026-08-12 14:56:02 on dist-test-slave-6zbq"
server_key: "206dee23cb24c7948e483d9778ac527a"
server_key_iv: "27584b8455dc21c72854997c8259d339"
server_key_version: "encryptionkey@0"
I20260812 14:56:02.541033 9865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "4873c2c4873c41498092ccc68ce5476e"
format_stamp: "Formatted at 2026-08-12 14:56:02 on dist-test-slave-6zbq"
server_key: "206dee23cb24c7948e483d9778ac527a"
server_key_iv: "27584b8455dc21c72854997c8259d339"
server_key_version: "encryptionkey@0"
I20260812 14:56:02.542271 9865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:56:02.542991 9890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:02.543141 9865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:02.543223 9865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "4873c2c4873c41498092ccc68ce5476e"
format_stamp: "Formatted at 2026-08-12 14:56:02 on dist-test-slave-6zbq"
server_key: "206dee23cb24c7948e483d9778ac527a"
server_key_iv: "27584b8455dc21c72854997c8259d339"
server_key_version: "encryptionkey@0"
I20260812 14:56:02.543285 9865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:02.553581 9865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:02.555166 9865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:02.555291 9865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:02.559661 9865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46819
I20260812 14:56:02.559691 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46819 every 8 connection(s)
I20260812 14:56:02.560002 9865 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:02.560914 9966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:02.562201 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 9865
I20260812 14:56:02.562295 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:02.562476 4204 external_mini_cluster.cc:1428] Setting key 0a47c409e10eedbea46217bd52867850
I20260812 14:56:02.563301 9966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e: Bootstrap starting.
I20260812 14:56:02.563876 9966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:02.564190 9966 log.cc:824] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:02.564884 9966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e: No bootstrap required, opened a new log
I20260812 14:56:02.567155 9966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4873c2c4873c41498092ccc68ce5476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260812 14:56:02.567310 9966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:02.567348 9966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4873c2c4873c41498092ccc68ce5476e, State: Initialized, Role: FOLLOWER
I20260812 14:56:02.567494 9966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4873c2c4873c41498092ccc68ce5476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260812 14:56:02.567593 9966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:02.567636 9966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:02.567685 9966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:02.568344 9966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4873c2c4873c41498092ccc68ce5476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
I20260812 14:56:02.568460 9966 leader_election.cc:302] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4873c2c4873c41498092ccc68ce5476e; no voters:
I20260812 14:56:02.568634 9966 leader_election.cc:288] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:02.568661 9975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:02.568766 9975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [term 1 LEADER]: Becoming Leader. State: Replica: 4873c2c4873c41498092ccc68ce5476e, State: Running, Role: LEADER
I20260812 14:56:02.568912 9966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:02.568926 9975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4873c2c4873c41498092ccc68ce5476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } }
Aug 12 14:56:02 dist-test-slave-6zbq krb5kdc[9840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:56:02.569298 9980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4873c2c4873c41498092ccc68ce5476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4873c2c4873c41498092ccc68ce5476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } } }
I20260812 14:56:02.569448 9980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:02.569326 9979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4873c2c4873c41498092ccc68ce5476e. Latest consensus state: current_term: 1 leader_uuid: "4873c2c4873c41498092ccc68ce5476e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4873c2c4873c41498092ccc68ce5476e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46819 } } }
I20260812 14:56:02.569599 9979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:02.569855 9997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:02.570953 9997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:02.572768 9997 catalog_manager.cc:1347] Generated new cluster ID: cb8980de498943419fea163611790e54
I20260812 14:56:02.572870 9997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:02.573201 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:02.562956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:43953 (local address 127.0.0.1:46819)
0812 14:56:02.563102 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:02.563110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:02.563133 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:56:02.563539 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:02.563546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:02.563594 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:02.563701 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:02.564481 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:02.565441 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:02.566530 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:02.566904 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:02.566977 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:02.569661 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:02.569665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:02.569666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:02.569677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:02.571584 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:02.572029 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:02.572032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:02.572032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:02.572132 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:02.572371 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:02.572372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:02.572372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:02.572577 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:02.572710 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:02.572880 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 14:56:02.572986 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260812 14:56:02.574069 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:02.579355 9997 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:02.579826 9997 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:02.587446 9997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4873c2c4873c41498092ccc68ce5476e: Generated new TSK 0
I20260812 14:56:02.587608 9997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:02.592727 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:02.578645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:52403 (local address 127.0.0.1:46819)
0812 14:56:02.578680 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:02.578682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:02.579059 (+ 377us) server_negotiation.cc:407] Connection header received
0812 14:56:02.579248 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:02.579249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:02.579260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:02.579314 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:02.587012 (+ 7698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:02.587775 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:02.588440 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:02.588783 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:02.588850 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:02.589884 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:02.589885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:02.589885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:02.589892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:02.591656 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:02.591999 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:02.592000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:02.592000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:02.592057 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:02.592265 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:02.592266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:02.592267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:02.592389 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:02.592491 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:02.592566 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:56:02.592633 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:56:02.594650 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 9865
W20260812 14:56:02.619884 9837 connection.cc:535] client connection to 127.0.0.1:46819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46819): Connection reset by peer (error 104)
2026-08-12T14:56:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:02 dist-test-slave-6zbq krb5kdc[10031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:02 dist-test-slave-6zbq krb5kdc[10031](info): set up 2 sockets
Aug 12 14:56:02 dist-test-slave-6zbq krb5kdc[10031](info): commencing operation
krb5kdc: starting...
W20260812 14:56:03.777058 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:03 dist-test-slave-6zbq krb5kdc[10031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:03.886593 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:03.963852 10053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:03.964044 10053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:03.964073 10053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:03.965174 10053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:03.965215 10053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:03.965243 10053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:03.965260 10053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:03.965282 10053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:03.966758 10053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:03.966989 10053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:03.967180 10053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:03.971522 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:03.971535 10064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:03.971556 10063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:03.971771 10053 server_base.cc:1034] running on GCE node
I20260812 14:56:03.971982 10053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:03.972251 10053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:03.973402 10053 hybrid_clock.cc:648] HybridClock initialized: now 1786546563973384 us; error 28 us; skew 500 ppm
Aug 12 14:56:03 dist-test-slave-6zbq krb5kdc[10031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:03.975708 10053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:03.976071 10053 webserver.cc:460] Webserver started at http://127.0.0.1:35005/ using document root <none> and password file <none>
I20260812 14:56:03.976271 10053 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:03.976325 10053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:03.976444 10053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:03.977391 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "be3dfd2421ef44aebd8e99fad3daba74"
format_stamp: "Formatted at 2026-08-12 14:56:03 on dist-test-slave-6zbq"
server_key: "93f18d5f8a2be8be4eb7ad51852c0f4e"
server_key_iv: "06422eef10be99e243ed488307a6b7ba"
server_key_version: "encryptionkey@0"
I20260812 14:56:03.977703 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "be3dfd2421ef44aebd8e99fad3daba74"
format_stamp: "Formatted at 2026-08-12 14:56:03 on dist-test-slave-6zbq"
server_key: "93f18d5f8a2be8be4eb7ad51852c0f4e"
server_key_iv: "06422eef10be99e243ed488307a6b7ba"
server_key_version: "encryptionkey@0"
I20260812 14:56:03.978896 10053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:56:03.979511 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:03.979655 10053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:03.979735 10053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "be3dfd2421ef44aebd8e99fad3daba74"
format_stamp: "Formatted at 2026-08-12 14:56:03 on dist-test-slave-6zbq"
server_key: "93f18d5f8a2be8be4eb7ad51852c0f4e"
server_key_iv: "06422eef10be99e243ed488307a6b7ba"
server_key_version: "encryptionkey@0"
I20260812 14:56:03.979791 10053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:04.000660 10053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:04.002197 10053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:04.002297 10053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:04.005738 10053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34847
I20260812 14:56:04.005767 10167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34847 every 8 connection(s)
I20260812 14:56:04.006050 10053 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:04.006958 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:04.009112 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74: Bootstrap starting.
I20260812 14:56:04.009613 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:04.009840 10169 log.cc:824] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:04.010398 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10053
I20260812 14:56:04.010500 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:04.010497 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74: No bootstrap required, opened a new log
I20260812 14:56:04.010665 4204 external_mini_cluster.cc:1428] Setting key b9dba775a001c294649d877baf062564
I20260812 14:56:04.013218 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3dfd2421ef44aebd8e99fad3daba74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260812 14:56:04.013377 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:04.013404 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3dfd2421ef44aebd8e99fad3daba74, State: Initialized, Role: FOLLOWER
I20260812 14:56:04.013517 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3dfd2421ef44aebd8e99fad3daba74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260812 14:56:04.013597 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:04.013640 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:04.013681 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:04.014322 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3dfd2421ef44aebd8e99fad3daba74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260812 14:56:04.014454 10169 leader_election.cc:302] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be3dfd2421ef44aebd8e99fad3daba74; no voters:
I20260812 14:56:04.014605 10169 leader_election.cc:288] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:04.014637 10177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:04.014780 10177 raft_consensus.cc:686] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [term 1 LEADER]: Becoming Leader. State: Replica: be3dfd2421ef44aebd8e99fad3daba74, State: Running, Role: LEADER
I20260812 14:56:04.014828 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:04.014940 10177 consensus_queue.cc:237] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3dfd2421ef44aebd8e99fad3daba74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } }
I20260812 14:56:04.015991 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be3dfd2421ef44aebd8e99fad3daba74. Latest consensus state: current_term: 1 leader_uuid: "be3dfd2421ef44aebd8e99fad3daba74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3dfd2421ef44aebd8e99fad3daba74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } } }
I20260812 14:56:04.016069 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:04.016245 10179 sys_catalog.cc:455] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be3dfd2421ef44aebd8e99fad3daba74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be3dfd2421ef44aebd8e99fad3daba74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34847 } } }
I20260812 14:56:04.016476 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:04.011121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:58715 (local address 127.0.0.1:34847)
0812 14:56:04.011333 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:04.011341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:04.011373 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:56:04.011817 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:04.011826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:04.011889 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:04.012034 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:04.012186 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:04.013506 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:04.014316 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:04.014934 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:04.015045 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:04.015698 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:04.015701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:04.015702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:04.015887 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:04.015927 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:04.015943 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:04.016023 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:04.016280 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 14:56:04.016307 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260812 14:56:04.016650 10179 sys_catalog.cc:458] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74 [sys.catalog]: This master's current role is: LEADER
W20260812 14:56:04.016568 10207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:04.016712 10207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:04.016778 10208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:04.016934 10208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:04.017539 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:04.018296 10208 catalog_manager.cc:1347] Generated new cluster ID: 7cc9a5954af540c39099464921195a56
I20260812 14:56:04.018365 10208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:04.024876 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:04.018870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:45361 (local address 127.0.0.1:34847)
0812 14:56:04.018902 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:04.018904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:04.022515 (+ 3611us) server_negotiation.cc:407] Connection header received
0812 14:56:04.022567 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:04.022568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:04.022577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:04.022633 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:04.022755 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:04.023195 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:04.023836 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:04.024048 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:04.024102 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:04.024660 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:04.024661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:04.024661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:04.024665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:04.024678 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:04.024682 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:04.024726 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:04.024802 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:56:04.024810 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:56:04.032289 10208 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:04.032961 10208 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:04.041465 10208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be3dfd2421ef44aebd8e99fad3daba74: Generated new TSK 0
I20260812 14:56:04.041644 10208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:04.043421 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10053
W20260812 14:56:04.059024 10023 connection.cc:535] client connection to 127.0.0.1:34847 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34847): Connection reset by peer (error 104)
2026-08-12T14:56:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:04 dist-test-slave-6zbq krb5kdc[10224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:04 dist-test-slave-6zbq krb5kdc[10224](info): set up 2 sockets
Aug 12 14:56:04 dist-test-slave-6zbq krb5kdc[10224](info): commencing operation
krb5kdc: starting...
W20260812 14:56:05.233146 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:05 dist-test-slave-6zbq krb5kdc[10224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:05.342941 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:05.424404 10241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:05.424604 10241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:05.424636 10241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:05.425740 10241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:05.425778 10241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:05.425797 10241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:05.425814 10241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:05.425829 10241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:05.427275 10241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:05.427510 10241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:05.427709 10241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:05.432296 10251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:05.432284 10252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:05.432287 10256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:05.432427 10241 server_base.cc:1034] running on GCE node
I20260812 14:56:05.432716 10241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:05.432963 10241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:05.434104 10241 hybrid_clock.cc:648] HybridClock initialized: now 1786546565434090 us; error 32 us; skew 500 ppm
Aug 12 14:56:05 dist-test-slave-6zbq krb5kdc[10224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:05.436389 10241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:05.436764 10241 webserver.cc:460] Webserver started at http://127.0.0.1:46309/ using document root <none> and password file <none>
I20260812 14:56:05.436991 10241 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:05.437039 10241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:05.437161 10241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:05.438110 10241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "8cfd3061b3124573a93e2e7c8ba2d748"
format_stamp: "Formatted at 2026-08-12 14:56:05 on dist-test-slave-6zbq"
server_key: "15a23ab4f34d177fac84fb40b9e2ac64"
server_key_iv: "4434cf5e415a1e011353502bc6afb5c7"
server_key_version: "encryptionkey@0"
I20260812 14:56:05.438460 10241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "8cfd3061b3124573a93e2e7c8ba2d748"
format_stamp: "Formatted at 2026-08-12 14:56:05 on dist-test-slave-6zbq"
server_key: "15a23ab4f34d177fac84fb40b9e2ac64"
server_key_iv: "4434cf5e415a1e011353502bc6afb5c7"
server_key_version: "encryptionkey@0"
I20260812 14:56:05.439615 10241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 14:56:05.440258 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:05.440405 10241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:56:05.440485 10241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "8cfd3061b3124573a93e2e7c8ba2d748"
format_stamp: "Formatted at 2026-08-12 14:56:05 on dist-test-slave-6zbq"
server_key: "15a23ab4f34d177fac84fb40b9e2ac64"
server_key_iv: "4434cf5e415a1e011353502bc6afb5c7"
server_key_version: "encryptionkey@0"
I20260812 14:56:05.440550 10241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:05.444468 10241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:05.446880 10241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:05.447019 10241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:05.450759 10241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41069
I20260812 14:56:05.450789 10370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41069 every 8 connection(s)
I20260812 14:56:05.451074 10241 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:05.451948 10371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:05.453991 10371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748: Bootstrap starting.
I20260812 14:56:05.454550 10371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:05.454783 10371 log.cc:824] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:05.455384 10371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748: No bootstrap required, opened a new log
I20260812 14:56:05.456846 10371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 14:56:05.456974 10371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:05.457015 10371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfd3061b3124573a93e2e7c8ba2d748, State: Initialized, Role: FOLLOWER
I20260812 14:56:05.457171 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10241
I20260812 14:56:05.457157 10371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 14:56:05.457242 10371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:05.457276 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:05.457281 10371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:05.457325 10371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:05.457495 4204 external_mini_cluster.cc:1428] Setting key 3f88109ed9673d5586aed16a93c8864e
I20260812 14:56:05.457917 10371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 14:56:05.458025 10371 leader_election.cc:302] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cfd3061b3124573a93e2e7c8ba2d748; no voters:
I20260812 14:56:05.458181 10371 leader_election.cc:288] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:05.458212 10382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:05.458416 10382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [term 1 LEADER]: Becoming Leader. State: Replica: 8cfd3061b3124573a93e2e7c8ba2d748, State: Running, Role: LEADER
I20260812 14:56:05.458441 10371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:05.458580 10382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } }
I20260812 14:56:05.459575 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:05.457926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:51395 (local address 127.0.0.1:41069)
0812 14:56:05.458101 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:05.458110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:05.458141 (+ 31us) server_negotiation.cc:407] Connection header received
0812 14:56:05.458252 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:05.458258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:05.458286 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:05.458708 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:05.459143 (+ 435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:05.459146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:05.459146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:05.459150 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:05.459190 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:05.459204 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:05.459272 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:05.459367 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 14:56:05.459408 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260812 14:56:05.459607 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cfd3061b3124573a93e2e7c8ba2d748. Latest consensus state: current_term: 1 leader_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } } }
I20260812 14:56:05.459703 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:05.460050 10383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cfd3061b3124573a93e2e7c8ba2d748" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41069 } } }
W20260812 14:56:05.460237 10409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 14:56:05.460371 10383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748 [sys.catalog]: This master's current role is: LEADER
W20260812 14:56:05.460379 10409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:05.460461 10410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:05.460634 10410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:05.461037 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:05.462193 10410 catalog_manager.cc:1347] Generated new cluster ID: f2b95fe852dd44faadbb863c99317d44
I20260812 14:56:05.462245 10410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:05.463442 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:05.462441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:46737 (local address 127.0.0.1:41069)
0812 14:56:05.462477 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:05.462479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:05.462650 (+ 171us) server_negotiation.cc:407] Connection header received
0812 14:56:05.462708 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:05.462709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:05.462715 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:05.462773 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:05.463249 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:05.463251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:05.463251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:05.463255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:05.463270 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:05.463274 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:05.463319 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:05.463369 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:56:05.463382 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:56:05.469192 10410 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:05.469341 10410 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:05.474494 10410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cfd3061b3124573a93e2e7c8ba2d748: Generated new TSK 0
I20260812 14:56:05.474671 10410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:05.477051 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10241
2026-08-12T14:56:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:05 dist-test-slave-6zbq krb5kdc[10434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:05 dist-test-slave-6zbq krb5kdc[10434](info): set up 2 sockets
Aug 12 14:56:05 dist-test-slave-6zbq krb5kdc[10434](info): commencing operation
krb5kdc: starting...
W20260812 14:56:06.652277 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:06 dist-test-slave-6zbq krb5kdc[10434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:06.760164 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:06.838500 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:06.838693 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:06.838722 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:06.839820 10455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:06.839861 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:06.839879 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:06.839895 10455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:06.839910 10455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:06.841379 10455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:06.841619 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:06.841820 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:06.846314 10465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:06.846323 10464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:06.846323 10468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:06.846568 10455 server_base.cc:1034] running on GCE node
I20260812 14:56:06.846796 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:06.847023 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:06.848153 10455 hybrid_clock.cc:648] HybridClock initialized: now 1786546566848139 us; error 30 us; skew 500 ppm
Aug 12 14:56:06 dist-test-slave-6zbq krb5kdc[10434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:06.850384 10455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:06.850783 10455 webserver.cc:460] Webserver started at http://127.0.0.1:35103/ using document root <none> and password file <none>
I20260812 14:56:06.850991 10455 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:06.851035 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:06.851153 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:06.852128 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "6df2199c912b4401aea93cb3813e4953"
format_stamp: "Formatted at 2026-08-12 14:56:06 on dist-test-slave-6zbq"
server_key: "162957cd072aff28b6d53f490eb80715"
server_key_iv: "8239ef63362787ba4d244494cc8906c2"
server_key_version: "encryptionkey@0"
I20260812 14:56:06.852452 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "6df2199c912b4401aea93cb3813e4953"
format_stamp: "Formatted at 2026-08-12 14:56:06 on dist-test-slave-6zbq"
server_key: "162957cd072aff28b6d53f490eb80715"
server_key_iv: "8239ef63362787ba4d244494cc8906c2"
server_key_version: "encryptionkey@0"
I20260812 14:56:06.853611 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:56:06.854204 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:06.854375 10455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:06.854461 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "6df2199c912b4401aea93cb3813e4953"
format_stamp: "Formatted at 2026-08-12 14:56:06 on dist-test-slave-6zbq"
server_key: "162957cd072aff28b6d53f490eb80715"
server_key_iv: "8239ef63362787ba4d244494cc8906c2"
server_key_version: "encryptionkey@0"
I20260812 14:56:06.854514 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:06.875402 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:06.877071 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:06.877172 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:06.880734 10455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43427
I20260812 14:56:06.880759 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43427 every 8 connection(s)
I20260812 14:56:06.881057 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:06.881965 10575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:06.884099 10575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953: Bootstrap starting.
I20260812 14:56:06.884382 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10455
I20260812 14:56:06.884469 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:06.884636 4204 external_mini_cluster.cc:1428] Setting key 3c037de72d00d5029cff156324922d3f
I20260812 14:56:06.884650 10575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:06.884953 10575 log.cc:824] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:06.885646 10575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953: No bootstrap required, opened a new log
I20260812 14:56:06.887779 10575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2199c912b4401aea93cb3813e4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43427 } }
I20260812 14:56:06.887940 10575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:06.887974 10575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df2199c912b4401aea93cb3813e4953, State: Initialized, Role: FOLLOWER
I20260812 14:56:06.888091 10575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2199c912b4401aea93cb3813e4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43427 } }
I20260812 14:56:06.888156 10575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:06.888190 10575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:06.888238 10575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:06.888782 10575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2199c912b4401aea93cb3813e4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43427 } }
I20260812 14:56:06.888870 10575 leader_election.cc:302] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df2199c912b4401aea93cb3813e4953; no voters:
I20260812 14:56:06.889019 10575 leader_election.cc:288] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:06.889060 10581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:06.889211 10581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [term 1 LEADER]: Becoming Leader. State: Replica: 6df2199c912b4401aea93cb3813e4953, State: Running, Role: LEADER
I20260812 14:56:06.889259 10575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:06.889387 10581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2199c912b4401aea93cb3813e4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43427 } }
I20260812 14:56:06.889758 10584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df2199c912b4401aea93cb3813e4953. Latest consensus state: current_term: 1 leader_uuid: "6df2199c912b4401aea93cb3813e4953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2199c912b4401aea93cb3813e4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43427 } } }
I20260812 14:56:06.889834 10584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:06.890262 10598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:06.890543 10589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df2199c912b4401aea93cb3813e4953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2199c912b4401aea93cb3813e4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43427 } } }
I20260812 14:56:06.890676 10589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:06.890785 10598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 14:56:06 dist-test-slave-6zbq krb5kdc[10434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 14:56:06.893338 10598 catalog_manager.cc:1347] Generated new cluster ID: f551fd1c875046fe9c7013db4bd183cb
I20260812 14:56:06.893388 10598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:06.894829 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:06.885091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:36365 (local address 127.0.0.1:43427)
0812 14:56:06.885255 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:06.885264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:06.885296 (+ 32us) server_negotiation.cc:407] Connection header received
0812 14:56:06.885749 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:06.885757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:06.885813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:06.885918 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:06.886657 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:06.887473 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:06.888459 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:06.888850 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:06.888945 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:06.891723 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:06.891727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:06.891727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:06.891739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:06.893196 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:06.893606 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:06.893610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:06.893610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:06.893722 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:06.893940 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:06.893941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:06.893941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:06.894252 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:06.894433 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:06.894524 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 14:56:06.894626 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"spinlock_wait_cycles":7936,"thread_start_us":99,"threads_started":1}
I20260812 14:56:06.895732 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:06.904946 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:06.897193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:37607 (local address 127.0.0.1:43427)
0812 14:56:06.898392 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:06.898394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:06.898402 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:56:06.898419 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:06.898420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:06.898427 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:06.898477 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:06.899086 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:06.899762 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:06.900716 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:06.901022 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:06.901075 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:06.902416 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:06.902417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:06.902418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:06.902424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:06.903810 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:06.904196 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:06.904198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:06.904198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:06.904270 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:06.904461 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:06.904462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:06.904462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:06.904585 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:06.904683 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:06.904788 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 14:56:06.904855 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181}
I20260812 14:56:06.934217 10598 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:06.934719 10598 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:06.939666 10598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df2199c912b4401aea93cb3813e4953: Generated new TSK 0
I20260812 14:56:06.939817 10598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:06.942807 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10455
W20260812 14:56:06.958542 10429 connection.cc:535] client connection to 127.0.0.1:43427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43427): Connection reset by peer (error 104)
2026-08-12T14:56:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:07 dist-test-slave-6zbq krb5kdc[10631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:07 dist-test-slave-6zbq krb5kdc[10631](info): set up 2 sockets
Aug 12 14:56:07 dist-test-slave-6zbq krb5kdc[10631](info): commencing operation
krb5kdc: starting...
W20260812 14:56:08.123495 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:08 dist-test-slave-6zbq krb5kdc[10631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:08.228644 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:08.305977 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:08.306159 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:08.306180 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:08.307261 10658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:08.307297 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:08.307310 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:08.307322 10658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:08.307331 10658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:08.308720 10658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43511
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:08.308902 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:08.309058 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:08.313371 10671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:08.313474 10658 server_base.cc:1034] running on GCE node
W20260812 14:56:08.313442 10666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:08.313393 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:08.313786 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:08.314057 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:08.315222 10658 hybrid_clock.cc:648] HybridClock initialized: now 1786546568315204 us; error 30 us; skew 500 ppm
Aug 12 14:56:08 dist-test-slave-6zbq krb5kdc[10631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:08.317538 10658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:08.317899 10658 webserver.cc:460] Webserver started at http://127.0.0.1:32863/ using document root <none> and password file <none>
I20260812 14:56:08.318071 10658 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:08.318110 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:08.318199 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:08.319216 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "c0ab1a05a065433fbb029087c856ef95"
format_stamp: "Formatted at 2026-08-12 14:56:08 on dist-test-slave-6zbq"
server_key: "b8e8d5440bd122bc685099bcaf716eae"
server_key_iv: "a9f5b77058fc1321926ae8b4eb95e6d1"
server_key_version: "encryptionkey@0"
I20260812 14:56:08.319504 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "c0ab1a05a065433fbb029087c856ef95"
format_stamp: "Formatted at 2026-08-12 14:56:08 on dist-test-slave-6zbq"
server_key: "b8e8d5440bd122bc685099bcaf716eae"
server_key_iv: "a9f5b77058fc1321926ae8b4eb95e6d1"
server_key_version: "encryptionkey@0"
I20260812 14:56:08.320653 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:56:08.321277 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:08.321411 10658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:08.321491 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "c0ab1a05a065433fbb029087c856ef95"
format_stamp: "Formatted at 2026-08-12 14:56:08 on dist-test-slave-6zbq"
server_key: "b8e8d5440bd122bc685099bcaf716eae"
server_key_iv: "a9f5b77058fc1321926ae8b4eb95e6d1"
server_key_version: "encryptionkey@0"
I20260812 14:56:08.321556 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:08.347651 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:08.349538 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:08.349660 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:08.353250 10658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43511
I20260812 14:56:08.353283 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43511 every 8 connection(s)
I20260812 14:56:08.353569 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:08.354316 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:08.356428 10763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95: Bootstrap starting.
I20260812 14:56:08.356943 10763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:08.357159 10763 log.cc:824] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:08.357774 10763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95: No bootstrap required, opened a new log
I20260812 14:56:08.359284 10763 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ab1a05a065433fbb029087c856ef95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260812 14:56:08.359413 10763 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:08.359441 10763 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ab1a05a065433fbb029087c856ef95, State: Initialized, Role: FOLLOWER
I20260812 14:56:08.359548 10763 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ab1a05a065433fbb029087c856ef95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260812 14:56:08.359623 10763 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:08.359653 10763 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:08.359684 10763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:08.360131 10763 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ab1a05a065433fbb029087c856ef95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260812 14:56:08.360213 10763 leader_election.cc:302] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ab1a05a065433fbb029087c856ef95; no voters:
I20260812 14:56:08.360332 10763 leader_election.cc:288] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:08.360373 10767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:08.360532 10767 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [term 1 LEADER]: Becoming Leader. State: Replica: c0ab1a05a065433fbb029087c856ef95, State: Running, Role: LEADER
I20260812 14:56:08.360538 10763 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:08.360678 10767 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ab1a05a065433fbb029087c856ef95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } }
I20260812 14:56:08.361027 10770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ab1a05a065433fbb029087c856ef95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ab1a05a065433fbb029087c856ef95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } } }
I20260812 14:56:08.361119 10770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:08.361142 10772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ab1a05a065433fbb029087c856ef95. Latest consensus state: current_term: 1 leader_uuid: "c0ab1a05a065433fbb029087c856ef95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ab1a05a065433fbb029087c856ef95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43511 } } }
I20260812 14:56:08.361193 10772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:08.361369 10785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:08.361891 10785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:08.362730 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10658
I20260812 14:56:08.362843 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:08.363032 4204 external_mini_cluster.cc:1428] Setting key 92c2ff6e21fb0896427ab396855b4484
I20260812 14:56:08.363283 10785 catalog_manager.cc:1347] Generated new cluster ID: 88dfb06ef0254aab82c9be913c4c1920
I20260812 14:56:08.363377 10785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:08.374328 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:08.363552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:50103 (local address 127.0.0.1:43511)
0812 14:56:08.370480 (+ 6928us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:08.370487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:08.370517 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:56:08.370606 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:08.370612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:08.370663 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:08.370792 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:08.371008 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:08.371919 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:08.372653 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:08.373047 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:08.373137 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:08.373735 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:08.373739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:08.373739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:08.373904 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:08.373933 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:08.373943 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:08.374011 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:08.374104 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 14:56:08.374131 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6878,"thread_start_us":76,"threads_started":1}
I20260812 14:56:08.375303 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:08.381150 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:08.378616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:56099 (local address 127.0.0.1:43511)
0812 14:56:08.378649 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:08.378651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:08.378773 (+ 122us) server_negotiation.cc:407] Connection header received
0812 14:56:08.378823 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:08.378823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:08.378833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:08.378877 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:08.378998 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:08.379451 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:08.380087 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:08.380303 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:08.380355 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:08.380917 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:08.380918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:08.380918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:08.380922 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:08.380936 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:08.380940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:08.380983 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:08.381059 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 14:56:08.381071 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:56:08.391208 10785 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:08.391670 10785 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:08.397332 10785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ab1a05a065433fbb029087c856ef95: Generated new TSK 0
I20260812 14:56:08.397495 10785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:08.399310 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10658
W20260812 14:56:08.410267 10628 connection.cc:535] client connection to 127.0.0.1:43511 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43511): Connection reset by peer (error 104)
2026-08-12T14:56:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:08 dist-test-slave-6zbq krb5kdc[10817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:08 dist-test-slave-6zbq krb5kdc[10817](info): set up 2 sockets
Aug 12 14:56:08 dist-test-slave-6zbq krb5kdc[10817](info): commencing operation
krb5kdc: starting...
W20260812 14:56:09.575982 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:09 dist-test-slave-6zbq krb5kdc[10817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 14:56:09.682123 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:09.759783 10837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:09.759964 10837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:09.759994 10837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:09.761073 10837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:09.761129 10837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:09.761149 10837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:09.761165 10837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:09.761181 10837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:09.762657 10837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:09.762877 10837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:09.763064 10837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:09.767370 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:09.767369 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:09.767365 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:09.767490 10837 server_base.cc:1034] running on GCE node
I20260812 14:56:09.767810 10837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:09.768054 10837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:09.769199 10837 hybrid_clock.cc:648] HybridClock initialized: now 1786546569769180 us; error 29 us; skew 500 ppm
Aug 12 14:56:09 dist-test-slave-6zbq krb5kdc[10817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:09.771425 10837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 14:56:09.771811 10837 webserver.cc:460] Webserver started at http://127.0.0.1:44925/ using document root <none> and password file <none>
I20260812 14:56:09.771971 10837 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:09.772008 10837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:09.772094 10837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:09.772980 10837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "dc786fd07af24dccae2aeff8d51a002e"
format_stamp: "Formatted at 2026-08-12 14:56:09 on dist-test-slave-6zbq"
server_key: "0420be634e17f8d459d13213031ab7bf"
server_key_iv: "6f3f9e042fba77452a96ba3964ed230d"
server_key_version: "encryptionkey@0"
I20260812 14:56:09.773263 10837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "dc786fd07af24dccae2aeff8d51a002e"
format_stamp: "Formatted at 2026-08-12 14:56:09 on dist-test-slave-6zbq"
server_key: "0420be634e17f8d459d13213031ab7bf"
server_key_iv: "6f3f9e042fba77452a96ba3964ed230d"
server_key_version: "encryptionkey@0"
I20260812 14:56:09.774420 10837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:56:09.775040 10868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:09.775182 10837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:09.775265 10837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "dc786fd07af24dccae2aeff8d51a002e"
format_stamp: "Formatted at 2026-08-12 14:56:09 on dist-test-slave-6zbq"
server_key: "0420be634e17f8d459d13213031ab7bf"
server_key_iv: "6f3f9e042fba77452a96ba3964ed230d"
server_key_version: "encryptionkey@0"
I20260812 14:56:09.775319 10837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:09.778266 10837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:09.780128 10837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:09.780229 10837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:09.783730 10837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35887
I20260812 14:56:09.783763 10961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35887 every 8 connection(s)
I20260812 14:56:09.784035 10837 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:09.784896 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:09.785802 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 10837
I20260812 14:56:09.785892 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:09.786051 4204 external_mini_cluster.cc:1428] Setting key 2e0a9449643dd2fe73fb183929309d95
I20260812 14:56:09.787267 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e: Bootstrap starting.
I20260812 14:56:09.787910 10962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:09.788213 10962 log.cc:824] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:09.788266 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:09.786501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:44543 (local address 127.0.0.1:35887)
0812 14:56:09.786654 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:09.786662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:09.786695 (+ 33us) server_negotiation.cc:407] Connection header received
0812 14:56:09.787153 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:09.787162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:09.787206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:09.787351 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:09.787836 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:09.787839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:09.787840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:09.787844 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:09.787888 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:09.787903 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:09.787996 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:09.788041 (+ 45us) server_negotiation.cc:299] Negotiation successful
0812 14:56:09.788083 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260812 14:56:09.788932 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e: No bootstrap required, opened a new log
I20260812 14:56:09.789086 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:09.790951 10962 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc786fd07af24dccae2aeff8d51a002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260812 14:56:09.791116 10962 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:09.791188 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:09.790256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:48395 (local address 127.0.0.1:35887)
0812 14:56:09.790287 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:09.790289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:09.790418 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:56:09.790490 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:09.790491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:09.790496 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:09.790550 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:09.791000 (+ 450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:09.791001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:09.791001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:09.791005 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:09.791018 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:09.791022 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:09.791074 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:09.791107 (+ 33us) server_negotiation.cc:299] Negotiation successful
0812 14:56:09.791118 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:56:09.791194 10962 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc786fd07af24dccae2aeff8d51a002e, State: Initialized, Role: FOLLOWER
I20260812 14:56:09.791365 10962 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc786fd07af24dccae2aeff8d51a002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260812 14:56:09.791455 10962 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:09.791478 10962 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:09.791524 10962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:09.792009 10962 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc786fd07af24dccae2aeff8d51a002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260812 14:56:09.792100 10962 leader_election.cc:302] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc786fd07af24dccae2aeff8d51a002e; no voters:
I20260812 14:56:09.792243 10962 leader_election.cc:288] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:09.792294 10979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:09.792433 10979 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [term 1 LEADER]: Becoming Leader. State: Replica: dc786fd07af24dccae2aeff8d51a002e, State: Running, Role: LEADER
I20260812 14:56:09.792459 10962 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:09.792519 10979 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc786fd07af24dccae2aeff8d51a002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260812 14:56:09.792857 10980 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc786fd07af24dccae2aeff8d51a002e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc786fd07af24dccae2aeff8d51a002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } } }
I20260812 14:56:09.793082 10980 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:09.793012 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc786fd07af24dccae2aeff8d51a002e. Latest consensus state: current_term: 1 leader_uuid: "dc786fd07af24dccae2aeff8d51a002e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc786fd07af24dccae2aeff8d51a002e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } } }
I20260812 14:56:09.793259 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:09.793475 10992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:09.793903 10992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:09.795357 10992 catalog_manager.cc:1347] Generated new cluster ID: 36c80b70b3644a1d84208b675bcaa956
I20260812 14:56:09.795398 10992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:09.809429 10992 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:09.809564 10992 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:09.813830 10992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc786fd07af24dccae2aeff8d51a002e: Generated new TSK 0
I20260812 14:56:09.813998 10992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:09.816807 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 10837
2026-08-12T14:56:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:09 dist-test-slave-6zbq krb5kdc[11009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:09 dist-test-slave-6zbq krb5kdc[11009](info): set up 2 sockets
Aug 12 14:56:09 dist-test-slave-6zbq krb5kdc[11009](info): commencing operation
krb5kdc: starting...
W20260812 14:56:11.004776 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:11 dist-test-slave-6zbq krb5kdc[11009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:11.112344 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:33383
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:39009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:33383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:11.193169 11029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:11.193369 11029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:11.193400 11029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:11.194540 11029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:11.194578 11029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:11.194597 11029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:11.194612 11029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:11.194636 11029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:11.196105 11029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:39009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:33383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:11.196429 11029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:11.196650 11029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:11.201493 11037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:11.201491 11038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:11.201484 11040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:11.201524 11029 server_base.cc:1034] running on GCE node
I20260812 14:56:11.202286 11029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:11.202607 11029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:11.203776 11029 hybrid_clock.cc:648] HybridClock initialized: now 1786546571203751 us; error 38 us; skew 500 ppm
Aug 12 14:56:11 dist-test-slave-6zbq krb5kdc[11009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546571, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:11.210991 11029 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:11.211400 11029 webserver.cc:460] Webserver started at http://127.4.27.62:39783/ using document root <none> and password file <none>
I20260812 14:56:11.211618 11029 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:11.211666 11029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:11.211793 11029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:11.212818 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "7c4c58664ea94d80adeb254d99862540"
format_stamp: "Formatted at 2026-08-12 14:56:11 on dist-test-slave-6zbq"
server_key: "09e23cb0c2f1fe26fc9fb3f08c4aae0b"
server_key_iv: "fb0fad246994f421a7c128b269857e30"
server_key_version: "encryptionkey@0"
I20260812 14:56:11.213156 11029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "7c4c58664ea94d80adeb254d99862540"
format_stamp: "Formatted at 2026-08-12 14:56:11 on dist-test-slave-6zbq"
server_key: "09e23cb0c2f1fe26fc9fb3f08c4aae0b"
server_key_iv: "fb0fad246994f421a7c128b269857e30"
server_key_version: "encryptionkey@0"
I20260812 14:56:11.214494 11029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 14:56:11.215250 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:11.215425 11029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:11.215517 11029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "7c4c58664ea94d80adeb254d99862540"
format_stamp: "Formatted at 2026-08-12 14:56:11 on dist-test-slave-6zbq"
server_key: "09e23cb0c2f1fe26fc9fb3f08c4aae0b"
server_key_iv: "fb0fad246994f421a7c128b269857e30"
server_key_version: "encryptionkey@0"
I20260812 14:56:11.215587 11029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:11.227356 11029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:11.228995 11029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:11.229120 11029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:11.232705 11029 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:33383
I20260812 14:56:11.232736 11154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:33383 every 8 connection(s)
I20260812 14:56:11.233037 11029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:11.233829 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:11.236019 11160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540: Bootstrap starting.
I20260812 14:56:11.236420 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11029
I20260812 14:56:11.236533 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:11.236567 11160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:11.236709 4204 external_mini_cluster.cc:1428] Setting key 23c8169ae8dbd40cd6b599daa6608421
I20260812 14:56:11.236835 11160 log.cc:824] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:11.237550 11160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540: No bootstrap required, opened a new log
I20260812 14:56:11.239696 11160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4c58664ea94d80adeb254d99862540" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33383 } }
I20260812 14:56:11.239861 11160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:11.239907 11160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4c58664ea94d80adeb254d99862540, State: Initialized, Role: FOLLOWER
I20260812 14:56:11.240026 11160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4c58664ea94d80adeb254d99862540" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33383 } }
I20260812 14:56:11.240110 11160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:11.240145 11160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:11.240186 11160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:11.240748 11160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4c58664ea94d80adeb254d99862540" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33383 } }
I20260812 14:56:11.240850 11160 leader_election.cc:302] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c4c58664ea94d80adeb254d99862540; no voters:
I20260812 14:56:11.241001 11160 leader_election.cc:288] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:11.241032 11167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:11.241197 11167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [term 1 LEADER]: Becoming Leader. State: Replica: 7c4c58664ea94d80adeb254d99862540, State: Running, Role: LEADER
I20260812 14:56:11.241279 11160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:11.241358 11167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4c58664ea94d80adeb254d99862540" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33383 } }
I20260812 14:56:11.241816 11167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c4c58664ea94d80adeb254d99862540. Latest consensus state: current_term: 1 leader_uuid: "7c4c58664ea94d80adeb254d99862540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4c58664ea94d80adeb254d99862540" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33383 } } }
I20260812 14:56:11.241966 11167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:11.241847 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c4c58664ea94d80adeb254d99862540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4c58664ea94d80adeb254d99862540" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 33383 } } }
I20260812 14:56:11.243099 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540 [sys.catalog]: This master's current role is: LEADER
Aug 12 14:56:11 dist-test-slave-6zbq krb5kdc[11009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:56:11.243695 11176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:11.244247 11176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:11.246790 11176 catalog_manager.cc:1347] Generated new cluster ID: edaf23ea62024317b2a03fbd613198c6
I20260812 14:56:11.246845 11176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:11.247620 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:11.237131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56011 (local address 127.4.27.62:33383)
0812 14:56:11.237278 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:11.237287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:11.237343 (+ 56us) server_negotiation.cc:407] Connection header received
0812 14:56:11.237810 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:11.237817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:11.237863 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:11.237966 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:11.238760 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:11.239569 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:11.240545 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:11.240985 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:11.241098 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:11.243976 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:11.243980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:11.243981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:11.243993 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:11.245601 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:11.246061 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:11.246064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:11.246065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:11.246174 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:11.246463 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:11.246464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:11.246465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:11.246748 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:11.246880 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:11.247336 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 14:56:11.247429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260812 14:56:11.248411 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:11.261476 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:11.250603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56163 (local address 127.4.27.62:33383)
0812 14:56:11.250637 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:11.250639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:11.254471 (+ 3832us) server_negotiation.cc:407] Connection header received
0812 14:56:11.254558 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:11.254559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:11.254570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:11.254630 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:11.255170 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:11.255947 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:11.256866 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:11.257200 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:11.257260 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:11.258870 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:11.258871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:11.258872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:11.258879 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:11.260354 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:11.260737 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:11.260738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:11.260738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:11.260792 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:11.260982 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:11.260982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:11.260982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:11.261115 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:11.261225 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:11.261328 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 14:56:11.261389 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:56:11.262688 11176 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:11.263165 11176 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:11.271670 11176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c4c58664ea94d80adeb254d99862540: Generated new TSK 0
I20260812 14:56:11.271828 11176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:11.278330 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11029
W20260812 14:56:11.296139 11006 connection.cc:535] client connection to 127.4.27.62:33383 recv error: Network error: failed to read from TLS socket (remote: 127.4.27.62:33383): Connection reset by peer (error 104)
2026-08-12T14:56:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:11 dist-test-slave-6zbq krb5kdc[11210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:11 dist-test-slave-6zbq krb5kdc[11210](info): set up 2 sockets
Aug 12 14:56:11 dist-test-slave-6zbq krb5kdc[11210](info): commencing operation
krb5kdc: starting...
W20260812 14:56:12.487790 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:12 dist-test-slave-6zbq krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:12.611647 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36657
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:36657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:12.721285 11236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:12.721464 11236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:12.721486 11236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:12.722613 11236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:12.722654 11236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:12.722666 11236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:12.722678 11236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:12.722707 11236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:12.724134 11236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:36657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:36657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:12.724355 11236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:12.724547 11236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:12.729043 11246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:12.729136 11249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:12.729032 11245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:12.729426 11236 server_base.cc:1034] running on GCE node
I20260812 14:56:12.729694 11236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:12.730015 11236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:12.731189 11236 hybrid_clock.cc:648] HybridClock initialized: now 1786546572731157 us; error 39 us; skew 500 ppm
Aug 12 14:56:12 dist-test-slave-6zbq krb5kdc[11210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546572, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:12.734079 11236 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:12.734668 11236 webserver.cc:460] Webserver started at http://127.4.27.62:41463/ using document root <none> and password file <none>
I20260812 14:56:12.734925 11236 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:12.734992 11236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:12.735131 11236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:12.736577 11236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "0a6dc26efe9143d6bc4dbc005a0fa430"
format_stamp: "Formatted at 2026-08-12 14:56:12 on dist-test-slave-6zbq"
server_key: "1069ecd6aea01c3c7588f986d9717160"
server_key_iv: "6d820477a464f2a44960b985e21c0b89"
server_key_version: "encryptionkey@0"
I20260812 14:56:12.736930 11236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "0a6dc26efe9143d6bc4dbc005a0fa430"
format_stamp: "Formatted at 2026-08-12 14:56:12 on dist-test-slave-6zbq"
server_key: "1069ecd6aea01c3c7588f986d9717160"
server_key_iv: "6d820477a464f2a44960b985e21c0b89"
server_key_version: "encryptionkey@0"
I20260812 14:56:12.738302 11236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:56:12.739027 11262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:12.739154 11236 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:12.739238 11236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "0a6dc26efe9143d6bc4dbc005a0fa430"
format_stamp: "Formatted at 2026-08-12 14:56:12 on dist-test-slave-6zbq"
server_key: "1069ecd6aea01c3c7588f986d9717160"
server_key_iv: "6d820477a464f2a44960b985e21c0b89"
server_key_version: "encryptionkey@0"
I20260812 14:56:12.739315 11236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:12.746223 11236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:12.748004 11236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:12.748122 11236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:12.751770 11236 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:36657
I20260812 14:56:12.751824 11357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:36657 every 8 connection(s)
I20260812 14:56:12.752116 11236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:12.752897 11358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:12.755119 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430: Bootstrap starting.
I20260812 14:56:12.755636 11358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:12.755888 11358 log.cc:824] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:12.756491 11358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430: No bootstrap required, opened a new log
I20260812 14:56:12.757369 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11236
I20260812 14:56:12.757465 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:12.757627 4204 external_mini_cluster.cc:1428] Setting key 3a43c6fc848a36165fa2d3acf35b5b4a
I20260812 14:56:12.758109 11358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36657 } }
I20260812 14:56:12.758266 11358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:12.758303 11358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6dc26efe9143d6bc4dbc005a0fa430, State: Initialized, Role: FOLLOWER
I20260812 14:56:12.758446 11358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36657 } }
I20260812 14:56:12.758522 11358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:12.758554 11358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:12.758601 11358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:12.759508 11358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36657 } }
I20260812 14:56:12.759609 11358 leader_election.cc:302] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a6dc26efe9143d6bc4dbc005a0fa430; no voters:
I20260812 14:56:12.759763 11358 leader_election.cc:288] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:12.759800 11367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:12.759899 11367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [term 1 LEADER]: Becoming Leader. State: Replica: 0a6dc26efe9143d6bc4dbc005a0fa430, State: Running, Role: LEADER
I20260812 14:56:12.760032 11358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:12.760044 11367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36657 } }
I20260812 14:56:12.760406 11367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a6dc26efe9143d6bc4dbc005a0fa430. Latest consensus state: current_term: 1 leader_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36657 } } }
I20260812 14:56:12.760640 11367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:12.760558 11369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6dc26efe9143d6bc4dbc005a0fa430" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 36657 } } }
I20260812 14:56:12.760876 11369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:12.761281 11385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:12.761817 11385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:12.763103 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:12.758054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44905 (local address 127.4.27.62:36657)
0812 14:56:12.758240 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:12.758248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:12.758271 (+ 23us) server_negotiation.cc:407] Connection header received
0812 14:56:12.758425 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:12.758433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:12.758477 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:12.758594 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:12.758829 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:12.759687 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:12.760713 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:12.761343 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:12.761429 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:12.762421 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:12.762425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:12.762425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:12.762592 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:12.762623 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:12.762634 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:12.762705 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:12.762871 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 14:56:12.762901 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260812 14:56:12.763852 11385 catalog_manager.cc:1347] Generated new cluster ID: 0adf4a9f9d714a3dbbd44fe4af737c79
I20260812 14:56:12.763931 11385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:12.764204 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:12.769014 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:12.765641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52955 (local address 127.4.27.62:36657)
0812 14:56:12.765677 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:12.765678 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:12.765826 (+ 148us) server_negotiation.cc:407] Connection header received
0812 14:56:12.765884 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:12.765884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:12.765894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:12.765941 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:12.766070 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:12.766783 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:12.767767 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:12.768067 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:12.768115 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:12.768773 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:12.768774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:12.768774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:12.768778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:12.768791 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:12.768796 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:12.768849 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:12.768932 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 14:56:12.768943 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260812 14:56:12.770201 11385 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:12.770723 11385 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:12.777442 11385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a6dc26efe9143d6bc4dbc005a0fa430: Generated new TSK 0
I20260812 14:56:12.777598 11385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:12.784904 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11236
W20260812 14:56:12.811331 11204 connection.cc:535] client connection to 127.4.27.62:36657 recv error: Network error: failed to read from TLS socket (remote: 127.4.27.62:36657): Connection reset by peer (error 104)
2026-08-12T14:56:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:12 dist-test-slave-6zbq krb5kdc[11419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:12 dist-test-slave-6zbq krb5kdc[11419](info): set up 2 sockets
Aug 12 14:56:12 dist-test-slave-6zbq krb5kdc[11419](info): commencing operation
krb5kdc: starting...
W20260812 14:56:13.979106 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:14 dist-test-slave-6zbq krb5kdc[11419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:14.084682 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:35637
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:35637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:14.163326 11440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:14.163530 11440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:14.163563 11440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:14.164669 11440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:14.164709 11440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:14.164728 11440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:14.164744 11440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:14.164759 11440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:14.166167 11440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:35637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:35637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:14.166529 11440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:14.166754 11440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:14.171133 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:14.171162 11440 server_base.cc:1034] running on GCE node
W20260812 14:56:14.171190 11450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:14.171113 11448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:14.171592 11440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:14.171866 11440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:14.173002 11440 hybrid_clock.cc:648] HybridClock initialized: now 1786546574172988 us; error 28 us; skew 500 ppm
Aug 12 14:56:14 dist-test-slave-6zbq krb5kdc[11419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546574, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:14.175287 11440 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:14.175661 11440 webserver.cc:460] Webserver started at http://127.4.27.62:33619/ using document root <none> and password file <none>
I20260812 14:56:14.175854 11440 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:14.175900 11440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:14.176019 11440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:14.176973 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "f2930ad7468f454b9ee3bd6c1be219c0"
format_stamp: "Formatted at 2026-08-12 14:56:14 on dist-test-slave-6zbq"
server_key: "cf16f495a1a1d94fbd2604fab728b876"
server_key_iv: "c697e2c24e4f244cec2857f3701244ff"
server_key_version: "encryptionkey@0"
I20260812 14:56:14.177287 11440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "f2930ad7468f454b9ee3bd6c1be219c0"
format_stamp: "Formatted at 2026-08-12 14:56:14 on dist-test-slave-6zbq"
server_key: "cf16f495a1a1d94fbd2604fab728b876"
server_key_iv: "c697e2c24e4f244cec2857f3701244ff"
server_key_version: "encryptionkey@0"
I20260812 14:56:14.178522 11440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:56:14.179148 11469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:14.179289 11440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:14.179371 11440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "f2930ad7468f454b9ee3bd6c1be219c0"
format_stamp: "Formatted at 2026-08-12 14:56:14 on dist-test-slave-6zbq"
server_key: "cf16f495a1a1d94fbd2604fab728b876"
server_key_iv: "c697e2c24e4f244cec2857f3701244ff"
server_key_version: "encryptionkey@0"
I20260812 14:56:14.179430 11440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:14.182508 11440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:14.184015 11440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:14.184137 11440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:14.188649 11440 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:35637
I20260812 14:56:14.188696 11561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:35637 every 8 connection(s)
I20260812 14:56:14.188990 11440 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:14.189867 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:14.191956 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0: Bootstrap starting.
I20260812 14:56:14.192441 11562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:14.192655 11562 log.cc:824] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:14.193234 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0: No bootstrap required, opened a new log
I20260812 14:56:14.194757 11562 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35637 } }
I20260812 14:56:14.194880 11562 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:14.194913 11562 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2930ad7468f454b9ee3bd6c1be219c0, State: Initialized, Role: FOLLOWER
I20260812 14:56:14.195013 11562 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35637 } }
I20260812 14:56:14.195084 11562 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:14.195111 11562 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:14.195142 11562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:14.195844 11562 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35637 } }
I20260812 14:56:14.195928 11562 leader_election.cc:302] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2930ad7468f454b9ee3bd6c1be219c0; no voters:
I20260812 14:56:14.196039 11562 leader_election.cc:288] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:14.196086 11573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:14.196244 11562 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:14.196260 11573 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [term 1 LEADER]: Becoming Leader. State: Replica: f2930ad7468f454b9ee3bd6c1be219c0, State: Running, Role: LEADER
I20260812 14:56:14.196417 11573 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35637 } }
I20260812 14:56:14.196758 11576 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2930ad7468f454b9ee3bd6c1be219c0. Latest consensus state: current_term: 1 leader_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35637 } } }
I20260812 14:56:14.196889 11576 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:14.196780 11575 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2930ad7468f454b9ee3bd6c1be219c0" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35637 } } }
I20260812 14:56:14.197052 11575 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:14.197219 11594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:14.197697 11594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:14.198490 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11440
I20260812 14:56:14.198608 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:14.198767 4204 external_mini_cluster.cc:1428] Setting key e53cdebf8b8bf365970c2ed09d02925c
I20260812 14:56:14.202741 11594 catalog_manager.cc:1347] Generated new cluster ID: 2c68b18877d746f3a4a4ff33a543616e
I20260812 14:56:14.202783 11594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:14.204171 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:14.199166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36561 (local address 127.4.27.62:35637)
0812 14:56:14.202813 (+ 3647us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:14.202821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:14.202850 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:56:14.202961 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:14.202967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:14.203014 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:14.203184 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:14.203797 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:14.203800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:14.203801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:14.203804 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:14.203845 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:14.203859 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:14.203919 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:14.203985 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 14:56:14.204019 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3605,"thread_start_us":68,"threads_started":1}
I20260812 14:56:14.204936 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:14.207284 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:14.206451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42405 (local address 127.4.27.62:35637)
0812 14:56:14.206483 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:14.206485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:14.206492 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:56:14.206522 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:14.206523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:14.206527 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:14.206588 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:14.207091 (+ 503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:14.207092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:14.207092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:14.207096 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:14.207110 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:14.207114 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:14.207171 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:14.207207 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:56:14.207226 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:56:14.229148 11594 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:14.229323 11594 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:14.235133 11594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2930ad7468f454b9ee3bd6c1be219c0: Generated new TSK 0
I20260812 14:56:14.235304 11594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:14.244753 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11440
2026-08-12T14:56:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:14 dist-test-slave-6zbq krb5kdc[11632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:14 dist-test-slave-6zbq krb5kdc[11632](info): set up 2 sockets
Aug 12 14:56:14 dist-test-slave-6zbq krb5kdc[11632](info): commencing operation
krb5kdc: starting...
W20260812 14:56:15.437582 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:15 dist-test-slave-6zbq krb5kdc[11632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:15.544848 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46475
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:46475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:15.622896 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:15.623077 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:15.623099 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:15.624191 11660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:15.624233 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:15.624245 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:15.624259 11660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:15.624284 11660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:15.625696 11660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:46475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:46475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:15.625901 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:15.626089 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:15.630462 11672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:15.630519 11660 server_base.cc:1034] running on GCE node
W20260812 14:56:15.630475 11670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:15.630587 11669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:15.630905 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:15.631162 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:15.632294 11660 hybrid_clock.cc:648] HybridClock initialized: now 1786546575632281 us; error 37 us; skew 500 ppm
Aug 12 14:56:15 dist-test-slave-6zbq krb5kdc[11632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546575, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:15.634610 11660 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:15.635004 11660 webserver.cc:460] Webserver started at http://127.4.27.62:33479/ using document root <none> and password file <none>
I20260812 14:56:15.635195 11660 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:15.635236 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:15.635365 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:15.636328 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "8cd6a2bc581c4320bb0868b84c39cefd"
format_stamp: "Formatted at 2026-08-12 14:56:15 on dist-test-slave-6zbq"
server_key: "6d8a342879f83f2c1eab411b0f54ecad"
server_key_iv: "d60ad35f16cbff0bbb74be59a0faacb6"
server_key_version: "encryptionkey@0"
I20260812 14:56:15.636641 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "8cd6a2bc581c4320bb0868b84c39cefd"
format_stamp: "Formatted at 2026-08-12 14:56:15 on dist-test-slave-6zbq"
server_key: "6d8a342879f83f2c1eab411b0f54ecad"
server_key_iv: "d60ad35f16cbff0bbb74be59a0faacb6"
server_key_version: "encryptionkey@0"
I20260812 14:56:15.637778 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:56:15.638424 11684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:15.638563 11660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:15.638645 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "8cd6a2bc581c4320bb0868b84c39cefd"
format_stamp: "Formatted at 2026-08-12 14:56:15 on dist-test-slave-6zbq"
server_key: "6d8a342879f83f2c1eab411b0f54ecad"
server_key_iv: "d60ad35f16cbff0bbb74be59a0faacb6"
server_key_version: "encryptionkey@0"
I20260812 14:56:15.638710 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:15.652467 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:15.654330 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:15.654524 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:15.658087 11660 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:46475
I20260812 14:56:15.658140 11792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:46475 every 8 connection(s)
I20260812 14:56:15.658447 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:15.658689 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11660
I20260812 14:56:15.658819 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:15.658998 4204 external_mini_cluster.cc:1428] Setting key 47a01e0253d2150634816b31257ec687
I20260812 14:56:15.659299 11794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:15.662175 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd: Bootstrap starting.
I20260812 14:56:15.662869 11794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:15.663165 11794 log.cc:824] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:15.663802 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd: No bootstrap required, opened a new log
Aug 12 14:56:15 dist-test-slave-6zbq krb5kdc[11632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:56:15.665565 11794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46475 } }
I20260812 14:56:15.665731 11794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:15.665783 11794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd6a2bc581c4320bb0868b84c39cefd, State: Initialized, Role: FOLLOWER
I20260812 14:56:15.665880 11794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46475 } }
I20260812 14:56:15.665949 11794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:15.665984 11794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:15.666033 11794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:15.666698 11794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46475 } }
I20260812 14:56:15.666802 11794 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd6a2bc581c4320bb0868b84c39cefd; no voters:
I20260812 14:56:15.666960 11794 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:15.667002 11800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:15.667148 11800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [term 1 LEADER]: Becoming Leader. State: Replica: 8cd6a2bc581c4320bb0868b84c39cefd, State: Running, Role: LEADER
I20260812 14:56:15.667210 11794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:15.667274 11800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46475 } }
I20260812 14:56:15.667685 11804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46475 } } }
I20260812 14:56:15.667760 11804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:15.667973 11800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd6a2bc581c4320bb0868b84c39cefd. Latest consensus state: current_term: 1 leader_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd6a2bc581c4320bb0868b84c39cefd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 46475 } } }
I20260812 14:56:15.668262 11800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:15.668414 11818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:15.668853 11818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:15.669324 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:15.659425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38897 (local address 127.4.27.62:46475)
0812 14:56:15.659608 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:15.659617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:15.659647 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:56:15.660299 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:15.660305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:15.660348 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:15.660452 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:15.661187 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:15.662012 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:15.663033 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:15.663438 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:15.663519 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:15.665528 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:15.665532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:15.665532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:15.665543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:15.667424 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:15.667833 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:15.667836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:15.667836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:15.667949 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:15.668164 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:15.668165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:15.668165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:15.668469 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:15.668946 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:15.669016 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 14:56:15.669143 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260812 14:56:15.670110 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:15.670419 11818 catalog_manager.cc:1347] Generated new cluster ID: 36d2dda51e43494080ca95df6a9cbd95
I20260812 14:56:15.670467 11818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:15.678561 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:15.671438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37507 (local address 127.4.27.62:46475)
0812 14:56:15.671491 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:15.671492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:15.671665 (+ 173us) server_negotiation.cc:407] Connection header received
0812 14:56:15.671720 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:15.671721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:15.671729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:15.671776 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:15.672341 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:15.673031 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:15.673956 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:15.674823 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:15.674883 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:15.676113 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:15.676114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:15.676115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:15.676121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:15.677509 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:15.677875 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:15.677875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:15.677876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:15.677923 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:15.678097 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:15.678098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:15.678098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:15.678198 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:15.678266 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:15.678421 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 14:56:15.678486 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:56:15.683444 11818 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:15.683879 11818 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:15.691424 11818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd6a2bc581c4320bb0868b84c39cefd: Generated new TSK 0
I20260812 14:56:15.691565 11818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:15.696527 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11660
2026-08-12T14:56:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:15 dist-test-slave-6zbq krb5kdc[11843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:15 dist-test-slave-6zbq krb5kdc[11843](info): set up 2 sockets
Aug 12 14:56:15 dist-test-slave-6zbq krb5kdc[11843](info): commencing operation
krb5kdc: starting...
W20260812 14:56:16.878237 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:16 dist-test-slave-6zbq krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:16.988551 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42197
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:17.069316 11868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:17.069506 11868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:17.069535 11868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:17.070644 11868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:17.070683 11868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:17.070701 11868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:17.070717 11868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:17.070734 11868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:17.072177 11868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:42197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:17.072403 11868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:17.072588 11868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:17.077420 11875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:17.077452 11877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:17.077458 11868 server_base.cc:1034] running on GCE node
W20260812 14:56:17.077426 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:17.077847 11868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:17.078094 11868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:17.079224 11868 hybrid_clock.cc:648] HybridClock initialized: now 1786546577079207 us; error 34 us; skew 500 ppm
Aug 12 14:56:17 dist-test-slave-6zbq krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546577, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:17.084597 11868 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:17.085001 11868 webserver.cc:460] Webserver started at http://127.4.27.62:45117/ using document root <none> and password file <none>
I20260812 14:56:17.085181 11868 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:17.085219 11868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:17.085305 11868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:17.086256 11868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "8d906b709f6e497899f662c875f3aeb2"
format_stamp: "Formatted at 2026-08-12 14:56:17 on dist-test-slave-6zbq"
server_key: "7fa02a63dc9b188b1fadcfef84b4fbf8"
server_key_iv: "bae098e32bc5a39443a087b8b22cabdf"
server_key_version: "encryptionkey@0"
I20260812 14:56:17.086608 11868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "8d906b709f6e497899f662c875f3aeb2"
format_stamp: "Formatted at 2026-08-12 14:56:17 on dist-test-slave-6zbq"
server_key: "7fa02a63dc9b188b1fadcfef84b4fbf8"
server_key_iv: "bae098e32bc5a39443a087b8b22cabdf"
server_key_version: "encryptionkey@0"
I20260812 14:56:17.087985 11868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:56:17.088826 11892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:17.088996 11868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:56:17.089082 11868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "8d906b709f6e497899f662c875f3aeb2"
format_stamp: "Formatted at 2026-08-12 14:56:17 on dist-test-slave-6zbq"
server_key: "7fa02a63dc9b188b1fadcfef84b4fbf8"
server_key_iv: "bae098e32bc5a39443a087b8b22cabdf"
server_key_version: "encryptionkey@0"
I20260812 14:56:17.089154 11868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:17.097451 11868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:17.099192 11868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:17.099309 11868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:17.102975 11868 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:42197
I20260812 14:56:17.103018 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:42197 every 8 connection(s)
I20260812 14:56:17.103309 11868 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:17.104092 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:17.106240 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2: Bootstrap starting.
I20260812 14:56:17.106822 11994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:17.107088 11994 log.cc:824] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:17.107970 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2: No bootstrap required, opened a new log
I20260812 14:56:17.109663 11994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d906b709f6e497899f662c875f3aeb2" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42197 } }
I20260812 14:56:17.109802 11994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:17.109838 11994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d906b709f6e497899f662c875f3aeb2, State: Initialized, Role: FOLLOWER
I20260812 14:56:17.109972 11994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d906b709f6e497899f662c875f3aeb2" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42197 } }
I20260812 14:56:17.110052 11994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:17.110095 11994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:17.110145 11994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:17.110968 11994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d906b709f6e497899f662c875f3aeb2" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42197 } }
I20260812 14:56:17.111065 11994 leader_election.cc:302] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d906b709f6e497899f662c875f3aeb2; no voters:
I20260812 14:56:17.111230 11994 leader_election.cc:288] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:17.111272 11999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:17.111404 11999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [term 1 LEADER]: Becoming Leader. State: Replica: 8d906b709f6e497899f662c875f3aeb2, State: Running, Role: LEADER
I20260812 14:56:17.111492 11994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:17.111546 11999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d906b709f6e497899f662c875f3aeb2" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42197 } }
I20260812 14:56:17.111933 12001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d906b709f6e497899f662c875f3aeb2. Latest consensus state: current_term: 1 leader_uuid: "8d906b709f6e497899f662c875f3aeb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d906b709f6e497899f662c875f3aeb2" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42197 } } }
I20260812 14:56:17.111927 12000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d906b709f6e497899f662c875f3aeb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d906b709f6e497899f662c875f3aeb2" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 42197 } } }
I20260812 14:56:17.112059 12001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:17.112068 12000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2 [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:17.112304 12014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:17.112823 12014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:17.112969 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 11868
I20260812 14:56:17.113081 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:17.113278 4204 external_mini_cluster.cc:1428] Setting key 558a0049f6b132a13587e5c5ae9ed1d2
I20260812 14:56:17.118103 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:17.113673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33751 (local address 127.4.27.62:42197)
0812 14:56:17.113824 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:17.113833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:17.113862 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:56:17.113975 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:17.113981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:17.114031 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:17.114137 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:17.114348 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:17.115224 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:17.116296 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:17.116675 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:17.116745 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:17.117353 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:17.117356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:17.117356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:17.117540 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:17.117577 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:17.117606 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:17.117671 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:17.117746 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 14:56:17.117899 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260812 14:56:17.118191 12014 catalog_manager.cc:1347] Generated new cluster ID: 9fb768f110384b9088b0af74e010d668
I20260812 14:56:17.118263 12014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:17.118992 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:17.122697 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:17.120216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44303 (local address 127.4.27.62:42197)
0812 14:56:17.120248 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:17.120250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:17.120343 (+ 93us) server_negotiation.cc:407] Connection header received
0812 14:56:17.120398 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:17.120399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:17.120409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:17.120449 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:17.120572 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:17.121015 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:17.121662 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:17.121861 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:17.121914 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:17.122497 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:17.122498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:17.122499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:17.122502 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:17.122515 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:17.122519 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:17.122562 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:17.122606 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 14:56:17.122631 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:56:17.135300 12014 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:17.135792 12014 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:17.144507 12014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d906b709f6e497899f662c875f3aeb2: Generated new TSK 0
I20260812 14:56:17.144671 12014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:17.149838 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 11868
2026-08-12T14:56:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:17 dist-test-slave-6zbq krb5kdc[12038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:17 dist-test-slave-6zbq krb5kdc[12038](info): set up 2 sockets
Aug 12 14:56:17 dist-test-slave-6zbq krb5kdc[12038](info): commencing operation
krb5kdc: starting...
W20260812 14:56:18.346845 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:18 dist-test-slave-6zbq krb5kdc[12038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:18.458277 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:35277
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:18.537420 12066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:18.537595 12066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:18.537616 12066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:18.538736 12066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:18.538775 12066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:18.538789 12066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:18.538812 12066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:18.538836 12066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:18.540316 12066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:18.540539 12066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:18.540737 12066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:18.545125 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:18.545233 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:18.545156 12076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:18.545328 12066 server_base.cc:1034] running on GCE node
I20260812 14:56:18.545529 12066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:18.545773 12066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:18.546906 12066 hybrid_clock.cc:648] HybridClock initialized: now 1786546578546893 us; error 30 us; skew 500 ppm
Aug 12 14:56:18 dist-test-slave-6zbq krb5kdc[12038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546578, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:18.549113 12066 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:18.549511 12066 webserver.cc:460] Webserver started at http://127.4.27.62:42801/ using document root <none> and password file <none>
I20260812 14:56:18.549726 12066 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:18.549768 12066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:18.549885 12066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:18.550850 12066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "e01fea14b63044388129fca0d93831cd"
format_stamp: "Formatted at 2026-08-12 14:56:18 on dist-test-slave-6zbq"
server_key: "4a699f8085879cafaea3efbcd6fbfe30"
server_key_iv: "f4d2173bd80fe2d84b83f46ccc01b776"
server_key_version: "encryptionkey@0"
I20260812 14:56:18.551148 12066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "e01fea14b63044388129fca0d93831cd"
format_stamp: "Formatted at 2026-08-12 14:56:18 on dist-test-slave-6zbq"
server_key: "4a699f8085879cafaea3efbcd6fbfe30"
server_key_iv: "f4d2173bd80fe2d84b83f46ccc01b776"
server_key_version: "encryptionkey@0"
I20260812 14:56:18.552328 12066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:56:18.552949 12091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:18.553095 12066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:18.553176 12066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "e01fea14b63044388129fca0d93831cd"
format_stamp: "Formatted at 2026-08-12 14:56:18 on dist-test-slave-6zbq"
server_key: "4a699f8085879cafaea3efbcd6fbfe30"
server_key_iv: "f4d2173bd80fe2d84b83f46ccc01b776"
server_key_version: "encryptionkey@0"
I20260812 14:56:18.553238 12066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:18.556317 12066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:18.557874 12066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:18.557996 12066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:18.561861 12066 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:35277
I20260812 14:56:18.561892 12189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:35277 every 8 connection(s)
I20260812 14:56:18.562170 12066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:18.563105 12190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:18.565048 12190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd: Bootstrap starting.
I20260812 14:56:18.565539 12190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:18.565755 12190 log.cc:824] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:18.566334 12190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd: No bootstrap required, opened a new log
I20260812 14:56:18.567806 12190 raft_consensus.cc:348] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01fea14b63044388129fca0d93831cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35277 } }
I20260812 14:56:18.567930 12190 raft_consensus.cc:374] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 14:56:18.567958 12190 raft_consensus.cc:729] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01fea14b63044388129fca0d93831cd, State: Initialized, Role: FOLLOWER
I20260812 14:56:18.568059 12190 consensus_queue.cc:260] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01fea14b63044388129fca0d93831cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35277 } }
I20260812 14:56:18.568131 12190 raft_consensus.cc:388] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 14:56:18.568159 12190 raft_consensus.cc:482] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 14:56:18.568189 12190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:18.568902 12190 raft_consensus.cc:504] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01fea14b63044388129fca0d93831cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35277 } }
I20260812 14:56:18.568989 12190 leader_election.cc:302] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e01fea14b63044388129fca0d93831cd; no voters:
I20260812 14:56:18.569103 12190 leader_election.cc:288] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 14:56:18.569156 12195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:18.569320 12190 sys_catalog.cc:564] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:18.569308 12195 raft_consensus.cc:686] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [term 1 LEADER]: Becoming Leader. State: Replica: e01fea14b63044388129fca0d93831cd, State: Running, Role: LEADER
I20260812 14:56:18.569429 12195 consensus_queue.cc:237] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01fea14b63044388129fca0d93831cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35277 } }
I20260812 14:56:18.569765 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e01fea14b63044388129fca0d93831cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01fea14b63044388129fca0d93831cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35277 } } }
I20260812 14:56:18.569842 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:18.570091 12209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:18.570039 12199 sys_catalog.cc:455] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e01fea14b63044388129fca0d93831cd. Latest consensus state: current_term: 1 leader_uuid: "e01fea14b63044388129fca0d93831cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e01fea14b63044388129fca0d93831cd" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 35277 } } }
I20260812 14:56:18.570333 12199 sys_catalog.cc:458] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:18.570645 12209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:18.572001 12209 catalog_manager.cc:1347] Generated new cluster ID: 76b8463ae08d45098d66287b1883ac33
I20260812 14:56:18.572044 12209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:18.572283 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12066
I20260812 14:56:18.572402 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:18.572580 4204 external_mini_cluster.cc:1428] Setting key 6043b5aaafadb6858489c596fcd1d41a
I20260812 14:56:18.574673 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:18.572987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42787 (local address 127.4.27.62:35277)
0812 14:56:18.573144 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:18.573153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:18.573252 (+ 99us) server_negotiation.cc:407] Connection header received
0812 14:56:18.573444 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:18.573450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:18.573486 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:18.573604 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:18.574266 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:18.574269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:18.574269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:18.574274 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:18.574312 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:18.574326 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:18.574403 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:18.574475 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:56:18.574511 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260812 14:56:18.575562 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:18.578078 12236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:18.577030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39533 (local address 127.4.27.62:35277)
0812 14:56:18.577073 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:18.577075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:18.577286 (+ 211us) server_negotiation.cc:407] Connection header received
0812 14:56:18.577314 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:18.577315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:18.577331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:18.577391 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:18.577889 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:18.577891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:18.577891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 14:56:18.577895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:18.577909 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 14:56:18.577914 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:18.577965 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:18.578012 (+ 47us) server_negotiation.cc:299] Negotiation successful
0812 14:56:18.578024 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260812 14:56:18.586818 12209 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:18.586957 12209 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:18.592188 12209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e01fea14b63044388129fca0d93831cd: Generated new TSK 0
I20260812 14:56:18.592332 12209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:18.596565 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12066
2026-08-12T14:56:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1444 ms)
[----------] 18 tests from AuthTokenIssuingTest (26076 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:18 dist-test-slave-6zbq krb5kdc[12250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:18 dist-test-slave-6zbq krb5kdc[12250](info): set up 2 sockets
Aug 12 14:56:18 dist-test-slave-6zbq krb5kdc[12250](info): commencing operation
krb5kdc: starting...
W20260812 14:56:19.777577 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:19 dist-test-slave-6zbq krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:19.886137 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39811
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39811,127.4.27.61:42989,127.4.27.60:39991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:19.965319 12278 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:56:19.965528 12278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:19.965544 12278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:19.965562 12278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:19.966741 12278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:19.966784 12278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:19.966796 12278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:19.966807 12278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:19.966817 12278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:19.968954 12278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39811,127.4.27.61:42989,127.4.27.60:39991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:39811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:19.969311 12278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:19.969551 12278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 14:56:19.974280 12278 server_base.cc:1034] running on GCE node
W20260812 14:56:19.974268 12283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:19.974287 12290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:19.974298 12284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:19.974742 12278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:19.975024 12278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:19.976162 12278 hybrid_clock.cc:648] HybridClock initialized: now 1786546579976148 us; error 33 us; skew 500 ppm
Aug 12 14:56:19 dist-test-slave-6zbq krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:19.978483 12278 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:19.978915 12278 webserver.cc:460] Webserver started at http://127.4.27.62:45593/ using document root <none> and password file <none>
I20260812 14:56:19.979117 12278 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:19.979166 12278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:19.979302 12278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:19.980321 12278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "b911439595844c84808715b256a8606e"
format_stamp: "Formatted at 2026-08-12 14:56:19 on dist-test-slave-6zbq"
server_key: "817d66ce8ed5c78261008ec2dc8ee98c"
server_key_iv: "ff63339d5c81478154ecb7a64007816d"
server_key_version: "encryptionkey@0"
I20260812 14:56:19.980651 12278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "b911439595844c84808715b256a8606e"
format_stamp: "Formatted at 2026-08-12 14:56:19 on dist-test-slave-6zbq"
server_key: "817d66ce8ed5c78261008ec2dc8ee98c"
server_key_iv: "ff63339d5c81478154ecb7a64007816d"
server_key_version: "encryptionkey@0"
I20260812 14:56:19.981869 12278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260812 14:56:19.982527 12300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:19.982661 12278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:19.982746 12278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "b911439595844c84808715b256a8606e"
format_stamp: "Formatted at 2026-08-12 14:56:19 on dist-test-slave-6zbq"
server_key: "817d66ce8ed5c78261008ec2dc8ee98c"
server_key_iv: "ff63339d5c81478154ecb7a64007816d"
server_key_version: "encryptionkey@0"
I20260812 14:56:19.982810 12278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:19.997293 12278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:19.998934 12278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:19.999060 12278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:20.002728 12278 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:39811
I20260812 14:56:20.002760 12408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:39811 every 8 connection(s)
I20260812 14:56:20.003055 12278 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:20.003928 12410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:20.005333 12410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:20.010298 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12278
I20260812 14:56:20.010414 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:56:20.010601 4204 external_mini_cluster.cc:1428] Setting key ab574ce4a4ffeda84b2aa4e8f6a4c3a6
I20260812 14:56:20.013657 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.005891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39811 (local address 127.0.0.1:56808)
0812 14:56:20.006043 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.006079 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.006539 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.006753 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.006755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.006971 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.007121 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.007126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.008027 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.008028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.008680 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.008682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.009091 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.009092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.009143 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.009529 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.009536 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.010896 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.012357 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.012363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.012365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.012539 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.012739 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.012739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.012740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.012825 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.013125 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.013129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.013190 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.013266 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.013435 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260812 14:56:20.013662 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.005933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.4.27.62:39811)
0812 14:56:20.006059 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.006067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.006139 (+ 72us) server_negotiation.cc:407] Connection header received
0812 14:56:20.006649 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.006651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.006667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.006741 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.007185 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.007956 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.008739 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.009020 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.009091 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.010970 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.010978 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.010978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.010985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.012281 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.012608 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.012611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.012611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.012683 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.012878 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.012878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.012879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.013065 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.013133 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.013310 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.013493 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260812 14:56:20.014493 12410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:20.014765 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.014670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50364)
0812 14:56:20.014700 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.014741 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260812 14:56:20.014882 12301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
I20260812 14:56:20.015261 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.015177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50376)
0812 14:56:20.015215 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.015238 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:56:20.015355 12410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:42989: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.27.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab.
Entry for principal kudu/127.4.27.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab.
I20260812 14:56:20.034231 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:42989
--webserver_interface=127.4.27.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab
--principal=kudu/127.4.27.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39811,127.4.27.61:42989,127.4.27.60:39991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:56:20.064560 12410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } attempt: 1
I20260812 14:56:20.065064 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.064930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50392)
0812 14:56:20.064991 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.065035 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 14:56:20.065418 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.065319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50394)
0812 14:56:20.065371 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.065394 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260812 14:56:20.065526 12410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:42989: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
W20260812 14:56:20.114600 12426 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:56:20.114789 12426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:20.114813 12426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:20.114845 12426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:20.115942 12426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:20.115983 12426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:20.116001 12426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:20.116016 12426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:20.116034 12426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:20.117494 12426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39811,127.4.27.61:42989,127.4.27.60:39991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab
--principal=kudu/127.4.27.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.27.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:20.117734 12426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:20.117933 12426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:20.122270 12436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:20.122270 12435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:20.122259 12438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:20.122550 12426 server_base.cc:1034] running on GCE node
I20260812 14:56:20.122783 12426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:20.123039 12426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:20.124187 12426 hybrid_clock.cc:648] HybridClock initialized: now 1786546580124173 us; error 30 us; skew 500 ppm
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:20.126820 12426 init.cc:377] Logged in from keytab as kudu/127.4.27.61@KRBTEST.COM (short username kudu)
I20260812 14:56:20.127239 12426 webserver.cc:460] Webserver started at http://127.4.27.61:44741/ using document root <none> and password file <none>
I20260812 14:56:20.127466 12426 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:20.127509 12426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:20.127627 12426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:20.128607 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data/instance:
uuid: "51920093cfeb4131b75ca8029c1ca686"
format_stamp: "Formatted at 2026-08-12 14:56:20 on dist-test-slave-6zbq"
server_key: "20bad7ed459b35b5f6caaaa2e435019f"
server_key_iv: "d1b3f557e138996a3ac4f0eebe79fb89"
server_key_version: "encryptionkey@0"
I20260812 14:56:20.128925 12426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal/instance:
uuid: "51920093cfeb4131b75ca8029c1ca686"
format_stamp: "Formatted at 2026-08-12 14:56:20 on dist-test-slave-6zbq"
server_key: "20bad7ed459b35b5f6caaaa2e435019f"
server_key_iv: "d1b3f557e138996a3ac4f0eebe79fb89"
server_key_version: "encryptionkey@0"
I20260812 14:56:20.130141 12426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260812 14:56:20.130827 12453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:20.130964 12426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:20.131047 12426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal
uuid: "51920093cfeb4131b75ca8029c1ca686"
format_stamp: "Formatted at 2026-08-12 14:56:20 on dist-test-slave-6zbq"
server_key: "20bad7ed459b35b5f6caaaa2e435019f"
server_key_iv: "d1b3f557e138996a3ac4f0eebe79fb89"
server_key_version: "encryptionkey@0"
I20260812 14:56:20.131111 12426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:20.133652 12410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } attempt: 2
I20260812 14:56:20.133976 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.133877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50396)
0812 14:56:20.133924 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.133952 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:56:20.134280 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.134194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50412)
0812 14:56:20.134233 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.134257 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260812 14:56:20.134369 12410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:42989: Network error: Client connection negotiation failed: client connection to 127.4.27.61:42989: connect: Connection refused (error 111)
I20260812 14:56:20.143419 12426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:20.145296 12426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:20.145406 12426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:20.149084 12426 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.61:42989
I20260812 14:56:20.149137 12542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.61:42989 every 8 connection(s)
I20260812 14:56:20.149552 12426 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
I20260812 14:56:20.149827 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12426
I20260812 14:56:20.149928 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-1/wal/instance
I20260812 14:56:20.150101 4204 external_mini_cluster.cc:1428] Setting key 0a90fdc76fb11f9fdce08088ce1f2bb5
I20260812 14:56:20.150602 12543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:20.151999 12543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.27.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.60@KRBTEST.COM" created.
I20260812 14:56:20.160602 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.152646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.4.27.62:39811)
0812 14:56:20.152698 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.152701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.152900 (+ 199us) server_negotiation.cc:407] Connection header received
0812 14:56:20.153337 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.153339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.153351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.153423 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.153966 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.154723 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.155657 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.155951 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.156028 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.158189 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.158191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.158191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.158197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.159405 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.159773 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.159773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.159774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.159833 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.160031 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.160032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.160032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.160125 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.160169 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.160476 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.160542 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 14:56:20.160813 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.152629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39811 (local address 127.0.0.1:56822)
0812 14:56:20.152784 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.152822 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.153245 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.153470 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.153473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.153692 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.153866 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.153870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.154807 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.154808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.155566 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.155568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.156036 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.156037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.156132 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.156553 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.156562 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.158102 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.159471 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.159478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.159479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.159682 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.159886 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.159887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.159887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.159980 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.160167 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.160170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.160340 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.160449 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.160603 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260812 14:56:20.161350 12543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:20.168888 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.161552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50420)
0812 14:56:20.161589 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.161598 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.161631 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.161868 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.161869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.161995 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.162040 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.162042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.163161 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.163161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.163881 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.163885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.164393 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.164394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.164684 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.165039 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.165044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.166450 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.167958 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.167959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.167960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.168110 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.168283 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.168284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.168284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.168337 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.168573 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.168573 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.168610 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.168671 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.168767 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 14:56:20.168964 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.161624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.4.27.61:42989)
0812 14:56:20.161733 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.161736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.161751 (+ 15us) server_negotiation.cc:407] Connection header received
0812 14:56:20.161767 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.161769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.161781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.161848 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.162104 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.163089 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.163964 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.164318 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.164370 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.166521 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.166529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.166530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.166536 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.167882 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.168176 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.168179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.168180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.168229 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.168385 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.168386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.168386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.168512 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.168580 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.168708 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.168836 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260812 14:56:20.169541 12543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:20.169847 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.169741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37040)
0812 14:56:20.169782 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.169826 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:56:20.169972 12456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
I20260812 14:56:20.170384 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.170247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37046)
0812 14:56:20.170318 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.170361 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:56:20.170477 12543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39991: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab.
Entry for principal kudu/127.4.27.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab.
I20260812 14:56:20.174937 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:39991
--webserver_interface=127.4.27.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab
--principal=kudu/127.4.27.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:39811,127.4.27.61:42989,127.4.27.60:39991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:56:20.219745 12543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } attempt: 1
I20260812 14:56:20.220425 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.220280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37054)
0812 14:56:20.220345 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.220393 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260812 14:56:20.220885 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.220785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37056)
0812 14:56:20.220832 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.220860 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 14:56:20.220989 12543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39991: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
I20260812 14:56:20.225497 12410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } attempt: 3
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:20.233789 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.225802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50430)
0812 14:56:20.225855 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.225868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.225923 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.226084 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.226085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.226245 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.226309 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.226312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.227298 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.227300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.228267 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.228278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.228920 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.228921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.229018 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.229521 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.229525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.231168 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.232848 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.232849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.232849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.233055 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.233248 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.233248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.233249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.233308 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.233544 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.233544 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.233591 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.233643 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.233723 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260812 14:56:20.233841 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.225847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.4.27.61:42989)
0812 14:56:20.225905 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.225907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.225920 (+ 13us) server_negotiation.cc:407] Connection header received
0812 14:56:20.225973 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.225974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.225987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.226047 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.226428 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.227188 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.228450 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.228800 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.228874 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.231273 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.231274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.231275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.231284 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.232770 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.233125 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.233126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.233126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.233191 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.233357 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.233358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.233358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.233491 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.233556 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.233678 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.233756 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:56:20.234369 12410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:20.234679 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.234590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37066)
0812 14:56:20.234629 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.234655 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260812 14:56:20.235010 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.234937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37070)
0812 14:56:20.234974 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.234995 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 14:56:20.235113 12410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39991: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
W20260812 14:56:20.257712 12556 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:56:20.257929 12556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:20.257946 12556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:20.257963 12556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:20.259029 12556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:20.259063 12556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:20.259080 12556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:20.259090 12556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:20.259109 12556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:20.260492 12556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:39811,127.4.27.61:42989,127.4.27.60:39991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab
--principal=kudu/127.4.27.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:39991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.27.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:20.260674 12556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:20.260836 12556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:20.265404 12573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:20.265493 12556 server_base.cc:1034] running on GCE node
W20260812 14:56:20.265408 12565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:20.265475 12567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:20.265829 12556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:20.266083 12556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:20.266238 12410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } attempt: 1
I20260812 14:56:20.266582 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.266482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37080)
0812 14:56:20.266528 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.266558 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260812 14:56:20.266875 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.266799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37096)
0812 14:56:20.266831 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.266854 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 14:56:20.266947 12410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39991: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
I20260812 14:56:20.267289 12556 hybrid_clock.cc:648] HybridClock initialized: now 1786546580267271 us; error 32 us; skew 500 ppm
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:20.269507 12556 init.cc:377] Logged in from keytab as kudu/127.4.27.60@KRBTEST.COM (short username kudu)
I20260812 14:56:20.269896 12556 webserver.cc:460] Webserver started at http://127.4.27.60:37711/ using document root <none> and password file <none>
I20260812 14:56:20.270066 12556 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:20.270108 12556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:20.270210 12556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:20.271170 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data/instance:
uuid: "010581db1b234ae8a7d13a6232efdb3b"
format_stamp: "Formatted at 2026-08-12 14:56:20 on dist-test-slave-6zbq"
server_key: "7b36eb45d2f8bea388219c3799e1dedd"
server_key_iv: "0a231eafc1e4de2768ef933d2bb1b9fb"
server_key_version: "encryptionkey@0"
I20260812 14:56:20.271459 12556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal/instance:
uuid: "010581db1b234ae8a7d13a6232efdb3b"
format_stamp: "Formatted at 2026-08-12 14:56:20 on dist-test-slave-6zbq"
server_key: "7b36eb45d2f8bea388219c3799e1dedd"
server_key_iv: "0a231eafc1e4de2768ef933d2bb1b9fb"
server_key_version: "encryptionkey@0"
I20260812 14:56:20.272583 12556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:56:20.273186 12584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:20.273324 12556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:20.273401 12556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal
uuid: "010581db1b234ae8a7d13a6232efdb3b"
format_stamp: "Formatted at 2026-08-12 14:56:20 on dist-test-slave-6zbq"
server_key: "7b36eb45d2f8bea388219c3799e1dedd"
server_key_iv: "0a231eafc1e4de2768ef933d2bb1b9fb"
server_key_version: "encryptionkey@0"
I20260812 14:56:20.273465 12556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:20.287766 12556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:20.289160 12543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } attempt: 2
I20260812 14:56:20.289536 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.289403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37102)
0812 14:56:20.289475 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.289510 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 14:56:20.289667 12556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:20.289819 12556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:20.289891 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.289809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37112)
0812 14:56:20.289844 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.289867 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:56:20.290027 12543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39991: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39991: connect: Connection refused (error 111)
I20260812 14:56:20.293606 12556 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.60:39991
I20260812 14:56:20.293646 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.60:39991 every 8 connection(s)
I20260812 14:56:20.293946 12556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
I20260812 14:56:20.294898 12676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:20.296314 12676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:20.299582 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12556
I20260812 14:56:20.299681 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786546313808590-4204-0/minicluster-data/master-2/wal/instance
I20260812 14:56:20.299868 4204 external_mini_cluster.cc:1428] Setting key 511cc16ff8d29489a20bb61db3cbf4f7
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:56:20.304733 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.296887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.4.27.62:39811)
0812 14:56:20.296960 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.296962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.297095 (+ 133us) server_negotiation.cc:407] Connection header received
0812 14:56:20.297523 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.297524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.297534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.297578 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.298092 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.298589 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.299524 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.299726 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.299787 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.302078 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.302079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.302079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.302086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.303448 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.303853 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.303854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.303855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.303905 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.304133 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.304135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.304135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.304237 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.304293 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.304571 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.304650 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 14:56:20.304883 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.296853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:39811 (local address 127.0.0.1:56834)
0812 14:56:20.296993 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.297025 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.297452 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.297637 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.297640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.297859 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.298035 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.298038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.298660 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.298661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.299454 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.299457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.299810 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.299812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.299960 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.300412 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.300431 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.302000 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.303524 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.303530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.303532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.303784 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.303966 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.303967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.303968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.304072 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.304293 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.304300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.304425 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.304531 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.304683 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:56:20.305560 12676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:20.308751 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.300204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.4.27.60:39991)
0812 14:56:20.300431 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.300435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.300658 (+ 223us) server_negotiation.cc:407] Connection header received
0812 14:56:20.300684 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.300687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.300732 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.300798 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.301597 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.302279 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.303175 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.303612 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.303668 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.305993 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.305999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.305999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.306004 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.307485 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.307807 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.307810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.307810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.307863 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.308029 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.308030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.308030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.308159 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.308301 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.308582 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.308655 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:20.311973 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.305766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:42989 (local address 127.0.0.1:50440)
0812 14:56:20.305805 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.305813 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.305846 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.306200 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.306201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.306286 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.306324 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.306325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.307023 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.307024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.307704 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.307706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.308032 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.308033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.308097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.308434 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.308438 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.309650 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.311129 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.311130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.311130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.311274 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.311444 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.311445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.311445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.311498 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.311748 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.311749 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.311789 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.311844 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.311916 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:56:20.312032 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.305844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.4.27.61:42989)
0812 14:56:20.305884 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.305886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.305893 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:56:20.305917 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.305918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.305928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.305988 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.306400 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.306950 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.307775 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.307972 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.308026 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.309709 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.309711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.309711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.309717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.311054 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.311337 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.311338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.311338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.311392 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.311546 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.311547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.311548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.311660 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.311720 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.311876 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.311950 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:56:20.312686 12676 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:56:20.316747 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.309459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.4.27.62:39811)
0812 14:56:20.309494 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.309496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.309507 (+ 11us) server_negotiation.cc:407] Connection header received
0812 14:56:20.309567 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.309567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.309577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.309631 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.310084 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.310737 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.311705 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.311948 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.312011 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.314400 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.314402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.314402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.314409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.315690 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.316036 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.316037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.316038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.316087 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.316258 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.316258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.316258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.316369 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.316488 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.316575 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.316649 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260812 14:56:20.318876 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.312900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37142)
0812 14:56:20.312947 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.312987 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.313039 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.313152 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.313153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.313281 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.313314 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.313316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.314151 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.314151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.314683 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.314686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.315107 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.315108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.315158 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.315439 (+ 281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.315443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.316558 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.317954 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.317955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.317955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.318159 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.318377 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.318377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.318378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.318435 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.318653 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.318653 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.318683 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.318726 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.318793 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 14:56:20.318940 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.312939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.4.27.60:39991)
0812 14:56:20.312969 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.312970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.313026 (+ 56us) server_negotiation.cc:407] Connection header received
0812 14:56:20.313093 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.313093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.313101 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.313139 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.313373 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.314083 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.314760 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.315042 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.315107 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.316633 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.316634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.316634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.316641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.317885 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.318222 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.318223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.318223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.318282 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.318485 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.318485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.318485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.318598 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.318668 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.318759 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.318852 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":18}
I20260812 14:56:20.320523 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: Bootstrap starting.
I20260812 14:56:20.321367 12676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: Neither blocks nor log segments found. Creating new log.
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:20.321677 12676 log.cc:824] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:20.322491 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: No bootstrap required, opened a new log
I20260812 14:56:20.324295 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.317336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.4.27.61:42989)
0812 14:56:20.317378 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.317380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.317549 (+ 169us) server_negotiation.cc:407] Connection header received
0812 14:56:20.317598 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.317599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.317609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.317660 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.318098 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.318755 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.319575 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.319844 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.319913 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.321999 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.322000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.322000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.322006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.323368 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.323651 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.323652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.323652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.323711 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.323842 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.323842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.323843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.323958 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.324091 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.324129 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.324213 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:56:20.324446 12676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.324559 12676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010581db1b234ae8a7d13a6232efdb3b, State: Initialized, Role: FOLLOWER
I20260812 14:56:20.324632 12676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.324928 12676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:20.324885 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.324992 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.325686 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 14:56:20.326963 12711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:20.327081 12711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:20.327023 12350 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b911439595844c84808715b256a8606e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56844
I20260812 14:56:20.326963 12556 master_runner.cc:186] Error getting master registration for 127.4.27.62:39811: OK, instance_id { permanent_uuid: "b911439595844c84808715b256a8606e" instance_seqno: 1786546580001707 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:56:20.329607 12556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:56:20.342022 12410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } attempt: 2
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546579, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:56:20.349052 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.342385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.4.27.60:39991)
0812 14:56:20.342457 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.342459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.342467 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:56:20.342488 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.342489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.342498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.342547 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.342811 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.343364 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.343988 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.344284 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.344347 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.346452 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.346454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.346454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.346461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.347953 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.348308 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.348309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.348309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.348361 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.348535 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.348535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.348536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.348692 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.348753 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.348889 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.348963 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 14:56:20.349057 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.342294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37144)
0812 14:56:20.342361 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.342373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.342413 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.342555 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.342556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.342703 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.342743 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.342745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.343428 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.343428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.343908 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.343909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.344347 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.344348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.344411 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.344870 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.344876 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.346311 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.348029 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.348029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.348030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.348240 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.348417 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.348418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.348419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.348482 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.348756 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.348757 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.348802 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.348869 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.348963 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:56:20.350672 12410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e: Bootstrap starting.
I20260812 14:56:20.351387 12410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:20.351671 12410 log.cc:824] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:20.352370 12410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e: No bootstrap required, opened a new log
I20260812 14:56:20.353325 12410 raft_consensus.cc:348] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.353432 12410 raft_consensus.cc:729] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b911439595844c84808715b256a8606e, State: Initialized, Role: FOLLOWER
I20260812 14:56:20.353508 12410 consensus_queue.cc:260] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.353742 12715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.353801 12410 sys_catalog.cc:564] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:20.353839 12715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.353919 12350 raft_consensus.cc:482] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 14:56:20.354025 12350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:20.354827 12350 raft_consensus.cc:504] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
W20260812 14:56:20.355561 12729 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:20.355612 12729 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:20.355665 12629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b911439595844c84808715b256a8606e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "010581db1b234ae8a7d13a6232efdb3b"
I20260812 14:56:20.355783 12629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:20.356624 12629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b911439595844c84808715b256a8606e in term 1.
I20260812 14:56:20.356884 12303 leader_election.cc:302] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 010581db1b234ae8a7d13a6232efdb3b, b911439595844c84808715b256a8606e; no voters:
I20260812 14:56:20.356995 12715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:20.357121 12715 raft_consensus.cc:686] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [term 1 LEADER]: Becoming Leader. State: Replica: b911439595844c84808715b256a8606e, State: Running, Role: LEADER
I20260812 14:56:20.357198 12715 consensus_queue.cc:237] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.357583 12715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b911439595844c84808715b256a8606e. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.357672 12715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:20.358011 12350 leader_election.cc:288] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [CANDIDATE]: Term 1 election: Requested vote from peers 51920093cfeb4131b75ca8029c1ca686 (127.4.27.61:42989), 010581db1b234ae8a7d13a6232efdb3b (127.4.27.60:39991)
I20260812 14:56:20.356289 12278 master_runner.cc:186] Error getting master registration for 127.4.27.61:42989: OK, instance_id { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" instance_seqno: 1786546580148024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:56:20.358495 12278 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:56:20.360117 12735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:20.368032 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.359413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.4.27.62:39811)
0812 14:56:20.359454 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.359456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.359855 (+ 399us) server_negotiation.cc:407] Connection header received
0812 14:56:20.359960 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.359961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.359972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.360001 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.361458 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.362174 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.363295 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.363576 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.363615 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.365191 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.365193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.365193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.365200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.366809 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.367207 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.367208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.367208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.367265 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.367467 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.367468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.367468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.367588 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.367657 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.367876 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.367950 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:56:20.371088 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.359535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.4.27.61:42989)
0812 14:56:20.359579 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.359581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.360857 (+ 1276us) server_negotiation.cc:407] Connection header received
0812 14:56:20.361114 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.361115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.361126 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.361156 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.362727 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.363463 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.364956 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.365279 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.365320 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.367514 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.367515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.367516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.367536 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.369447 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.370006 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.370007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.370008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.370075 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.370401 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.370402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.370402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.370539 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.370607 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.370906 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.370996 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:56:20.373178 12735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:20.375067 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.362563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.4.27.60:39991)
0812 14:56:20.362612 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.362614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.368655 (+ 6041us) server_negotiation.cc:407] Connection header received
0812 14:56:20.368743 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.368744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.368754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.368794 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.369331 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.370047 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.370839 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.371045 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.371083 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.372311 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.372312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.372312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.372319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.373758 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.374188 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.374190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.374190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.374249 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.374490 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.374491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.374491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.374613 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.374693 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.374900 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.374978 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260812 14:56:20.375171 12629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [term 1 FOLLOWER]: Refusing update from remote peer b911439595844c84808715b256a8606e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:56:20.375536 12733 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [LEADER]: Connected to new peer: Peer: permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:56:20.376889 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b911439595844c84808715b256a8606e. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.376984 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.378541 12733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.378638 12733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:20.378741 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.378832 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.379007 12747 mvcc.cc:205] Tried to move back new op lower bound from 7317694793213530112 to 7317694793143775232. Current Snapshot: MvccSnapshot[applied={T|T < 7317694793213530112}]
I20260812 14:56:20.379532 12749 mvcc.cc:205] Tried to move back new op lower bound from 7317694793213530112 to 7317694793143775232. Current Snapshot: MvccSnapshot[applied={T|T < 7317694793213530112}]
I20260812 14:56:20.381168 12543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } attempt: 3
I20260812 14:56:20.382773 12735 catalog_manager.cc:1347] Generated new cluster ID: f2401b3ade5a45ea93305f60707f03f0
I20260812 14:56:20.382824 12735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 12 14:56:20 dist-test-slave-6zbq krb5kdc[12250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546580, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:56:20.388725 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.381397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39991 (local address 127.0.0.1:37170)
0812 14:56:20.381440 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:20.381449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:20.381481 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:20.381716 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:20.381717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:20.381881 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:20.381920 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.381922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.382867 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.382868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.383621 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:20.383623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.384319 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.384319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:20.384377 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.384775 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:20.384779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:20.386108 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:20.387749 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.387767 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.387767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.387984 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.388168 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:20.388169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:20.388169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:20.388228 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:20.388462 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:20.388462 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:20.388501 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 14:56:20.388553 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 14:56:20.388637 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:56:20.389047 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:20.381547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.4.27.60:39991)
0812 14:56:20.381602 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.381604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.381611 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:56:20.381631 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.381632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.381641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.381701 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.381994 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.382774 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.383709 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:20.383994 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:20.384042 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:20.386190 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:20.386191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:20.386192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:20.386198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:20.387687 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.388051 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.388052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.388052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.388101 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:20.388286 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:20.388287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:20.388287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:20.388393 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:20.388440 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:20.388902 (+ 462us) server_negotiation.cc:299] Negotiation successful
0812 14:56:20.388967 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:56:20.390312 12543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: Bootstrap starting.
I20260812 14:56:20.391021 12543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:20.391296 12543 log.cc:824] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:20.392221 12543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: No bootstrap required, opened a new log
I20260812 14:56:20.394101 12543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.394225 12543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51920093cfeb4131b75ca8029c1ca686, State: Initialized, Role: FOLLOWER
I20260812 14:56:20.394304 12543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } }
I20260812 14:56:20.395077 12497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b911439595844c84808715b256a8606e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51920093cfeb4131b75ca8029c1ca686"
I20260812 14:56:20.395177 12497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [term 0 FOLLOWER]: Advancing to term 1
W20260812 14:56:20.395915 12462 tablet.cc:2369] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 14:56:20.395974 12497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b911439595844c84808715b256a8606e in term 1.
I20260812 14:56:20.396348 12496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [term 1 FOLLOWER]: Refusing update from remote peer b911439595844c84808715b256a8606e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:56:20.396591 12543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:20.396584 12733 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:56:20.397974 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.398075 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.398128 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b911439595844c84808715b256a8606e. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.398176 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.398207 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.398250 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:20.399582 12735 catalog_manager.cc:1370] Generated new certificate authority record
W20260812 14:56:20.399952 12791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:20.399997 12791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:20.400238 12735 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:20.406563 12735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e: Generated new TSK 0
I20260812 14:56:20.406721 12735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 14:56:20.421250 12770 mvcc.cc:205] Tried to move back new op lower bound from 7317694793284042752 to 7317694793143775232. Current Snapshot: MvccSnapshot[applied={T|T < 7317694793213530112}]
I20260812 14:56:20.421761 12715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b911439595844c84808715b256a8606e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b911439595844c84808715b256a8606e" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 39811 } } peers { permanent_uuid: "51920093cfeb4131b75ca8029c1ca686" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 42989 } } peers { permanent_uuid: "010581db1b234ae8a7d13a6232efdb3b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39991 } } }
I20260812 14:56:20.421866 12715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b911439595844c84808715b256a8606e [sys.catalog]: This master's current role is: LEADER
W20260812 14:56:20.436586 12419 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.435277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.4.27.62:39811)
0812 14:56:20.435319 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.435321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.435968 (+ 647us) server_negotiation.cc:407] Connection header received
0812 14:56:20.436065 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.436066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.436077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.436107 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.436522 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56866: BlockingRecv error: recv got EOF from 127.0.0.1:56866 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:56:20.437285 12679 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.435695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.4.27.60:39991)
0812 14:56:20.435734 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.435735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.436818 (+ 1083us) server_negotiation.cc:407] Connection header received
0812 14:56:20.436907 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.436908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.436919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.436947 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.437226 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37180: BlockingRecv error: recv got EOF from 127.0.0.1:37180 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:56:20.438207 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.437592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.4.27.61:42989)
0812 14:56:20.437630 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.437632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.437751 (+ 119us) server_negotiation.cc:407] Connection header received
0812 14:56:20.437835 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.437836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.437846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.437873 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.438149 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50470: BlockingRecv error: recv got EOF from 127.0.0.1:50470 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260812 14:56:20.440357 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.439671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.4.27.61:42989)
0812 14:56:20.439718 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.439719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.439878 (+ 159us) server_negotiation.cc:407] Connection header received
0812 14:56:20.439975 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.439975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.439992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.440019 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.440314 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50478: BlockingRecv error: recv got EOF from 127.0.0.1:50478 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260812 14:56:20.443216 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.442542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.4.27.61:42989)
0812 14:56:20.442573 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.442574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.442783 (+ 209us) server_negotiation.cc:407] Connection header received
0812 14:56:20.442869 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.442870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.442879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.442905 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.443177 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50492: BlockingRecv error: recv got EOF from 127.0.0.1:50492 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:56:20.451038 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.446501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.4.27.61:42989)
0812 14:56:20.446533 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.446535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.450528 (+ 3993us) server_negotiation.cc:407] Connection header received
0812 14:56:20.450625 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.450626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.450636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.450663 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.450992 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50506: BlockingRecv error: recv got EOF from 127.0.0.1:50506 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:56:20.459426 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.458636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.4.27.61:42989)
0812 14:56:20.458670 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.458672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.458934 (+ 262us) server_negotiation.cc:407] Connection header received
0812 14:56:20.459025 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.459026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.459036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.459062 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.459381 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50508: BlockingRecv error: recv got EOF from 127.0.0.1:50508 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 14:56:20.469501 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.468466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.4.27.61:42989)
0812 14:56:20.468498 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.468500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.469012 (+ 512us) server_negotiation.cc:407] Connection header received
0812 14:56:20.469113 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.469114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.469126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.469157 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.469456 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50510: BlockingRecv error: recv got EOF from 127.0.0.1:50510 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 14:56:20.487946 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.486880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.4.27.61:42989)
0812 14:56:20.486919 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.486921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.487359 (+ 438us) server_negotiation.cc:407] Connection header received
0812 14:56:20.487468 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.487469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.487482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.487511 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.487894 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50514: BlockingRecv error: recv got EOF from 127.0.0.1:50514 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260812 14:56:20.522253 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.521481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.4.27.61:42989)
0812 14:56:20.521528 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.521530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.521692 (+ 162us) server_negotiation.cc:407] Connection header received
0812 14:56:20.521747 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.521748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.521762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.521820 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.522193 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50524: BlockingRecv error: recv got EOF from 127.0.0.1:50524 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:56:20.588119 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.587426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.4.27.61:42989)
0812 14:56:20.587514 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.587517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.587588 (+ 71us) server_negotiation.cc:407] Connection header received
0812 14:56:20.587663 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.587664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.587676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.587727 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.588055 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50534: BlockingRecv error: recv got EOF from 127.0.0.1:50534 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 14:56:20.717830 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.717199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.4.27.61:42989)
0812 14:56:20.717276 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.717278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.717306 (+ 28us) server_negotiation.cc:407] Connection header received
0812 14:56:20.717386 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.717387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.717399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.717447 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.717767 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50548: BlockingRecv error: recv got EOF from 127.0.0.1:50548 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 14:56:20.975440 12550 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:20.974856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.4.27.61:42989)
0812 14:56:20.974931 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:20.974933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:20.974943 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:56:20.974995 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:20.974995 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:20.975007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:20.975054 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:20.975379 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50550: BlockingRecv error: recv got EOF from 127.0.0.1:50550 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 14:56:21.327811 12711 catalog_manager.cc:1259] Loaded cluster ID: f2401b3ade5a45ea93305f60707f03f0
I20260812 14:56:21.327879 12711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: loading cluster ID for follower catalog manager: success
I20260812 14:56:21.328662 12711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: acquiring CA information for follower catalog manager: success
I20260812 14:56:21.328946 12711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 010581db1b234ae8a7d13a6232efdb3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:56:21.400390 12791 catalog_manager.cc:1259] Loaded cluster ID: f2401b3ade5a45ea93305f60707f03f0
I20260812 14:56:21.400456 12791 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: loading cluster ID for follower catalog manager: success
I20260812 14:56:21.401245 12791 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: acquiring CA information for follower catalog manager: success
I20260812 14:56:21.401535 12791 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51920093cfeb4131b75ca8029c1ca686: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:56:21.490926 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:21.488461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.4.27.61:42989)
0812 14:56:21.488579 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:21.488581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:21.488603 (+ 22us) server_negotiation.cc:407] Connection header received
0812 14:56:21.488683 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:21.488684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:21.488742 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:21.488811 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:21.488942 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.489440 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.490173 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.490399 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.490455 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:21.490601 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:21.490714 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:21.490782 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:21.490819 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 14:56:21.490841 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260812 14:56:21.494410 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:21.492441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.4.27.60:39991)
0812 14:56:21.492530 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:21.492532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:21.492539 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:56:21.492574 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:21.492575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:21.492606 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:21.492670 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:21.492754 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.493212 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.493830 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.494002 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.494040 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:21.494152 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:21.494233 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:21.494285 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:21.494300 (+ 15us) server_negotiation.cc:299] Negotiation successful
0812 14:56:21.494314 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":44,"threads_started":1}
I20260812 14:56:21.498524 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:21.495936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.4.27.60:39991)
0812 14:56:21.495987 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:21.495989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:21.495995 (+ 6us) server_negotiation.cc:407] Connection header received
0812 14:56:21.496017 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:21.496017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:21.496026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:21.496074 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:21.496183 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.496929 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.497821 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.498016 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.498069 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:21.498271 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:21.498303 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:21.498388 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:21.498440 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 14:56:21.498462 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 14:56:21.498973 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:21.495957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.4.27.61:42989)
0812 14:56:21.495988 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:21.495990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:21.495999 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:56:21.496052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:21.496053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:21.496062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:21.496104 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:21.496224 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.496938 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.498076 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.498333 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.498410 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:21.498677 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:21.498706 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:21.498759 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:21.498887 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 14:56:21.498911 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260812 14:56:21.499136 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:21.495758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.4.27.62:39811)
0812 14:56:21.496027 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:21.496029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:21.496237 (+ 208us) server_negotiation.cc:407] Connection header received
0812 14:56:21.496280 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:21.496281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:21.496317 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:21.496376 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:21.496490 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.496998 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.498097 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:21.498443 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:21.498495 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:21.498783 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:21.498875 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:21.498940 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:21.499033 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 14:56:21.499060 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":44,"threads_started":1}
I20260812 14:56:21.500545 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12278
I20260812 14:56:21.513866 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12426
I20260812 14:56:21.525033 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12556
2026-08-12T14:56:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2922 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 14:56:21 dist-test-slave-6zbq krb5kdc[12943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 14:56:21 dist-test-slave-6zbq krb5kdc[12943](info): set up 2 sockets
Aug 12 14:56:21 dist-test-slave-6zbq krb5kdc[12943](info): commencing operation
krb5kdc: starting...
W20260812 14:56:22.699172 4204 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/test-user.keytab.
Aug 12 14:56:22 dist-test-slave-6zbq krb5kdc[12943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T14:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T14:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.27.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
Entry for principal kudu/127.4.27.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab.
I20260812 14:56:22.806908 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41785
--webserver_interface=127.4.27.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:41785,127.4.27.61:36087,127.4.27.60:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 14:56:22.884799 12968 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:56:22.884980 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:22.885004 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:22.885028 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:22.886097 12968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:22.886132 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:22.886152 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:22.886168 12968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:22.886183 12968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:22.887615 12968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:41785,127.4.27.61:36087,127.4.27.60:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.62.keytab
--principal=kudu/127.4.27.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.62:41785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.27.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:22.887846 12968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:22.888032 12968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:22.892459 12979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:22.892443 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:22.892494 12968 server_base.cc:1034] running on GCE node
W20260812 14:56:22.892459 12981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:22.892951 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:22.893177 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:22.894363 12968 hybrid_clock.cc:648] HybridClock initialized: now 1786546582894328 us; error 28 us; skew 500 ppm
Aug 12 14:56:22 dist-test-slave-6zbq krb5kdc[12943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:22.900736 12968 init.cc:377] Logged in from keytab as kudu/127.4.27.62@KRBTEST.COM (short username kudu)
I20260812 14:56:22.901131 12968 webserver.cc:460] Webserver started at http://127.4.27.62:44527/ using document root <none> and password file <none>
I20260812 14:56:22.901316 12968 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:22.901361 12968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:22.901479 12968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:22.902469 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data/instance:
uuid: "b5a9b98763c04503ae64118e2efcea3a"
format_stamp: "Formatted at 2026-08-12 14:56:22 on dist-test-slave-6zbq"
server_key: "2473416c9dda12557010e23dc3b58300"
server_key_iv: "bc4fe3c9774628b296e0ca1b4d074113"
server_key_version: "encryptionkey@0"
I20260812 14:56:22.902788 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal/instance:
uuid: "b5a9b98763c04503ae64118e2efcea3a"
format_stamp: "Formatted at 2026-08-12 14:56:22 on dist-test-slave-6zbq"
server_key: "2473416c9dda12557010e23dc3b58300"
server_key_iv: "bc4fe3c9774628b296e0ca1b4d074113"
server_key_version: "encryptionkey@0"
I20260812 14:56:22.903961 12968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 14:56:22.904595 12998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:22.904739 12968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260812 14:56:22.904821 12968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal
uuid: "b5a9b98763c04503ae64118e2efcea3a"
format_stamp: "Formatted at 2026-08-12 14:56:22 on dist-test-slave-6zbq"
server_key: "2473416c9dda12557010e23dc3b58300"
server_key_iv: "bc4fe3c9774628b296e0ca1b4d074113"
server_key_version: "encryptionkey@0"
I20260812 14:56:22.904881 12968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:22.913458 12968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:22.915099 12968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:22.915222 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:22.918840 12968 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.62:41785
I20260812 14:56:22.918867 13081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.62:41785 every 8 connection(s)
I20260812 14:56:22.919159 12968 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/data/info.pb
I20260812 14:56:22.920185 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:22.920605 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 12968
I20260812 14:56:22.920681 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-0/wal/instance
I20260812 14:56:22.920853 4204 external_mini_cluster.cc:1428] Setting key 0e596b46b7f0387f5a3ac817e99fa92a
I20260812 14:56:22.921623 13086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 14:56:22 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.27.61@KRBTEST.COM; defaulting to no policy
I20260812 14:56:22.930248 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:22.922419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.4.27.62:41785)
0812 14:56:22.922538 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:22.922542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:22.922572 (+ 30us) server_negotiation.cc:407] Connection header received
0812 14:56:22.923102 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:22.923104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:22.923119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:22.923189 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:22.923785 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:22.924542 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:22.925391 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:22.925691 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:22.925781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:22.927587 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:22.927593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:22.927594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:22.927600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:22.928940 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:22.929311 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:22.929314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:22.929314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:22.929397 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:22.929586 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:22.929587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:22.929588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:22.929776 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:22.929837 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:22.929993 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 14:56:22.930080 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260812 14:56:22.930373 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:22.922208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41785 (local address 127.0.0.1:43682)
0812 14:56:22.922397 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:22.922440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:22.922959 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:22.923222 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:22.923225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:22.923537 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:22.923698 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:22.923702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:22.924628 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:22.924628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:22.925318 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:22.925321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:22.925775 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:22.925776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:22.925836 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:22.926273 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:22.926280 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:22.927515 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:22.929009 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:22.929015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:22.929017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:22.929239 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:22.929454 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:22.929455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:22.929455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:22.929532 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:22.929837 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:22.929842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:22.929898 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 14:56:22.929968 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 14:56:22.930156 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":71,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.61@KRBTEST.COM" created.
I20260812 14:56:22.931005 13086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:22.931334 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:22.931222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47764)
0812 14:56:22.931265 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:22.931313 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:56:22.931473 12999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
I20260812 14:56:22.931829 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:22.931744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47768)
0812 14:56:22.931783 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:22.931810 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260812 14:56:22.931927 13086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:36087: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab.
Entry for principal kudu/127.4.27.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab.
I20260812 14:56:22.942298 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:36087
--webserver_interface=127.4.27.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab
--principal=kudu/127.4.27.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:41785,127.4.27.61:36087,127.4.27.60:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:56:22.981115 13086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } attempt: 1
I20260812 14:56:22.981467 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:22.981347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47782)
0812 14:56:22.981400 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:22.981440 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 14:56:22.981752 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:22.981679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47794)
0812 14:56:22.981711 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:22.981735 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260812 14:56:22.981840 13086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:36087: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
W20260812 14:56:23.019310 13100 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:56:23.019480 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:23.019496 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:23.019531 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:23.020592 13100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:23.020634 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:23.020648 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:23.020670 13100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:23.020694 13100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:23.022130 13100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:41785,127.4.27.61:36087,127.4.27.60:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.61.keytab
--principal=kudu/127.4.27.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.61:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.27.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:23.022351 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:23.022536 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:23.026693 13108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:23.026703 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:23.026703 13113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:23.026894 13100 server_base.cc:1034] running on GCE node
I20260812 14:56:23.027096 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:23.027355 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:23.028492 13100 hybrid_clock.cc:648] HybridClock initialized: now 1786546583028479 us; error 25 us; skew 500 ppm
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:23.030710 13100 init.cc:377] Logged in from keytab as kudu/127.4.27.61@KRBTEST.COM (short username kudu)
I20260812 14:56:23.031045 13100 webserver.cc:460] Webserver started at http://127.4.27.61:42221/ using document root <none> and password file <none>
I20260812 14:56:23.031221 13100 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:23.031260 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:23.031343 13100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:23.032233 13100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data/instance:
uuid: "0bd6c7644a374db9a042be6fa1599b6c"
format_stamp: "Formatted at 2026-08-12 14:56:23 on dist-test-slave-6zbq"
server_key: "64ad01019bad99a3ec5642f07a571867"
server_key_iv: "78586157a6a867f94aadab42ee6b56d7"
server_key_version: "encryptionkey@0"
I20260812 14:56:23.032500 13100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal/instance:
uuid: "0bd6c7644a374db9a042be6fa1599b6c"
format_stamp: "Formatted at 2026-08-12 14:56:23 on dist-test-slave-6zbq"
server_key: "64ad01019bad99a3ec5642f07a571867"
server_key_iv: "78586157a6a867f94aadab42ee6b56d7"
server_key_version: "encryptionkey@0"
I20260812 14:56:23.033574 13100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 14:56:23.034142 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:23.034291 13100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:23.034391 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal
uuid: "0bd6c7644a374db9a042be6fa1599b6c"
format_stamp: "Formatted at 2026-08-12 14:56:23 on dist-test-slave-6zbq"
server_key: "64ad01019bad99a3ec5642f07a571867"
server_key_iv: "78586157a6a867f94aadab42ee6b56d7"
server_key_version: "encryptionkey@0"
I20260812 14:56:23.034449 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:23.049974 13086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } attempt: 2
I20260812 14:56:23.050282 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.050180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47810)
0812 14:56:23.050224 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.050257 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:56:23.050586 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.050508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47814)
0812 14:56:23.050546 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.050569 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:56:23.050664 13086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.61:36087: Network error: Client connection negotiation failed: client connection to 127.4.27.61:36087: connect: Connection refused (error 111)
I20260812 14:56:23.058306 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:23.059805 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:23.059924 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:23.063426 13100 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.61:36087
I20260812 14:56:23.063453 13226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.61:36087 every 8 connection(s)
I20260812 14:56:23.063735 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/data/info.pb
I20260812 14:56:23.064590 13228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:23.065982 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13100
I20260812 14:56:23.066082 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-1/wal/instance
I20260812 14:56:23.065975 13228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:23.066285 4204 external_mini_cluster.cc:1428] Setting key 4e872b2bb187b389c67c68da507d324d
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.27.60@KRBTEST.COM; defaulting to no policy
I20260812 14:56:23.074755 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.066647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.4.27.62:41785)
0812 14:56:23.066691 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.066693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.066899 (+ 206us) server_negotiation.cc:407] Connection header received
0812 14:56:23.067424 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.067425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.067435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.067472 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.068157 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.068829 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.069708 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.069983 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.070064 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.072112 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.072113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.072113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.072128 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.073550 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.073909 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.073910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.073910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.073966 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.074163 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.074163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.074164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.074290 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.074383 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.074638 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.074702 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260812 14:56:23.074875 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.066632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41785 (local address 127.0.0.1:43688)
0812 14:56:23.066779 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.066819 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.067341 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.067536 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.067539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.067836 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.068085 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.068090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.068904 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.068905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.069632 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.069634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.070068 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.070069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.070160 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.070627 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.070635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.072021 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.073627 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.073634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.073636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.073837 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.074028 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.074028 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.074028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.074114 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.074386 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.074393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.074514 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.074602 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.074711 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260812 14:56:23.075363 13228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.27.60@KRBTEST.COM" created.
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:23.081959 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.075561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47818)
0812 14:56:23.075610 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.075620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.075656 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.075843 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.075843 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.075966 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.076006 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.076007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.076888 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.076889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.077361 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.077362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.077840 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.077841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.077913 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.078258 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.078261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.079414 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.080952 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.080953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.080954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.081178 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.081352 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.081353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.081353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.081414 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.081628 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.081629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.081671 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.081742 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.081833 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 14:56:23.082012 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.075589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.4.27.61:36087)
0812 14:56:23.075722 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.075726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.075745 (+ 19us) server_negotiation.cc:407] Connection header received
0812 14:56:23.075762 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.075764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.075773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.075837 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.076135 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.076814 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.077424 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.077751 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.077821 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.079507 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.079514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.079514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.079520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.080875 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.081249 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.081251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.081251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.081297 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.081464 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.081465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.081465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.081573 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.081630 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.081764 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.081884 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260812 14:56:23.082517 13228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:23.082808 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.082694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44096)
0812 14:56:23.082738 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.082784 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 14:56:23.082942 13133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
I20260812 14:56:23.083338 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.083241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44104)
0812 14:56:23.083287 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.083315 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260812 14:56:23.083436 13228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39001: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.27.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab.
Entry for principal kudu/127.4.27.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab.
I20260812 14:56:23.089969 4204 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktWKW5U/build/release/bin/kudu
/tmp/dist-test-tasktWKW5U/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:39001
--webserver_interface=127.4.27.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab
--principal=kudu/127.4.27.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.27.20:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.27.62:41785,127.4.27.61:36087,127.4.27.60:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 14:56:23.132624 13228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } attempt: 1
I20260812 14:56:23.132989 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.132888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44114)
0812 14:56:23.132932 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.132965 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 14:56:23.133255 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.133182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44126)
0812 14:56:23.133215 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.133238 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260812 14:56:23.133339 13228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39001: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
I20260812 14:56:23.141798 13086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } attempt: 3
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:23.148144 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.142061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47826)
0812 14:56:23.142103 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.142113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.142162 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.142279 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.142280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.142438 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.142495 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.142496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.143361 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.143361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.143848 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.143850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.144307 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.144308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.144357 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.144753 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.144758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.145851 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.147379 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.147379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.147380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.147528 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.147694 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.147695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.147695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.147738 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.147942 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.147943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.147980 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.148023 (+ 43us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.148088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:56:23.148204 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.142109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.4.27.61:36087)
0812 14:56:23.142163 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.142165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.142194 (+ 29us) server_negotiation.cc:407] Connection header received
0812 14:56:23.142215 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.142216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.142227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.142275 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.142557 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.143287 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.143919 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.144244 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.144311 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.145931 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.145933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.145933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.145940 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.147311 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.147584 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.147585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.147585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.147639 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.147781 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.147782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.147782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.147896 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.147953 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.148045 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.148116 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 14:56:23.148543 13086 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:23.148803 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.148725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44132)
0812 14:56:23.148764 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.148787 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 14:56:23.149178 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.149109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44138)
0812 14:56:23.149140 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.149162 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260812 14:56:23.149281 13086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39001: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
W20260812 14:56:23.168913 13245 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 14:56:23.169075 13245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 14:56:23.169091 13245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 14:56:23.169114 13245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 14:56:23.170133 13245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 14:56:23.170164 13245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 14:56:23.170176 13245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 14:56:23.170186 13245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 14:56:23.170199 13245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 14:56:23.171582 13245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.27.20:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.27.62:41785,127.4.27.61:36087,127.4.27.60:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/krb5kdc/kudu_127.4.27.60.keytab
--principal=kudu/127.4.27.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.27.60:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.27.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 12 Aug 2026 13:51:05 UTC on ccycloud.quasar-hjnkdx.root.comops.site
build id 4582765
I20260812 14:56:23.171756 13245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 14:56:23.171911 13245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 14:56:23.176050 13258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:23.176056 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 14:56:23.176047 13261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 14:56:23.176227 13245 server_base.cc:1034] running on GCE node
I20260812 14:56:23.176468 13245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 14:56:23.176714 13245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 14:56:23.177847 13245 hybrid_clock.cc:648] HybridClock initialized: now 1786546583177834 us; error 24 us; skew 500 ppm
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 14:56:23.179858 13245 init.cc:377] Logged in from keytab as kudu/127.4.27.60@KRBTEST.COM (short username kudu)
I20260812 14:56:23.180230 13245 webserver.cc:460] Webserver started at http://127.4.27.60:40305/ using document root <none> and password file <none>
I20260812 14:56:23.180393 13086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } attempt: 1
I20260812 14:56:23.180428 13245 fs_manager.cc:359] Metadata directory not provided
I20260812 14:56:23.180470 13245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 14:56:23.180585 13245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 14:56:23.180658 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.180571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44152)
0812 14:56:23.180608 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.180635 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260812 14:56:23.180932 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.180860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44160)
0812 14:56:23.180891 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.180912 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260812 14:56:23.181010 13086 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.27.60:39001: Network error: Client connection negotiation failed: client connection to 127.4.27.60:39001: connect: Connection refused (error 111)
I20260812 14:56:23.181604 13245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data/instance:
uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b"
format_stamp: "Formatted at 2026-08-12 14:56:23 on dist-test-slave-6zbq"
server_key: "5fe2e608c7ca64e24ca8a04864dca8e0"
server_key_iv: "00586ba54a6a12170aee12dc4bc50df4"
server_key_version: "encryptionkey@0"
I20260812 14:56:23.181901 13245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal/instance:
uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b"
format_stamp: "Formatted at 2026-08-12 14:56:23 on dist-test-slave-6zbq"
server_key: "5fe2e608c7ca64e24ca8a04864dca8e0"
server_key_iv: "00586ba54a6a12170aee12dc4bc50df4"
server_key_version: "encryptionkey@0"
I20260812 14:56:23.183104 13245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260812 14:56:23.183751 13274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 14:56:23.183888 13245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260812 14:56:23.183975 13245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data,/tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal
uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b"
format_stamp: "Formatted at 2026-08-12 14:56:23 on dist-test-slave-6zbq"
server_key: "5fe2e608c7ca64e24ca8a04864dca8e0"
server_key_iv: "00586ba54a6a12170aee12dc4bc50df4"
server_key_version: "encryptionkey@0"
I20260812 14:56:23.184038 13245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 14:56:23.194674 13245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 14:56:23.196146 13245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 14:56:23.196260 13245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 14:56:23.199828 13245 rpc_server.cc:307] RPC server started. Bound to: 127.4.27.60:39001
I20260812 14:56:23.199867 13367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.27.60:39001 every 8 connection(s)
I20260812 14:56:23.200156 13245 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/data/info.pb
I20260812 14:56:23.200924 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 14:56:23.201490 13228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } attempt: 2
I20260812 14:56:23.202490 13368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:23.203707 4204 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktWKW5U/build/release/bin/kudu as pid 13245
I20260812 14:56:23.203809 4204 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktWKW5U/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786546313808590-4204-0/minicluster-data/master-2/wal/instance
I20260812 14:56:23.204018 4204 external_mini_cluster.cc:1428] Setting key 75c8cc22ede04ec866828a624ef682ca
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.61@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:56:23.210966 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.203098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.4.27.62:41785)
0812 14:56:23.203143 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.203145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.203274 (+ 129us) server_negotiation.cc:407] Connection header received
0812 14:56:23.203333 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.203334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.203344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.203390 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.203959 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.204549 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.205371 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.205592 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.205645 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.207943 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.207945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.207945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.207952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.209575 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.209983 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.209984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.209984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.210042 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.210316 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.210317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.210318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.210463 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.210523 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.210812 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.210885 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260812 14:56:23.211203 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.201758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44168)
0812 14:56:23.201812 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.201824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.201874 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.202625 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.202626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.202787 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.202834 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.202836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.204647 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.204648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.205387 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.205389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.205961 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.205962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.206016 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.206389 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.206394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.208315 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.210169 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.210170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.210170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.210395 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.210595 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.210596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.210596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.210664 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.210897 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.210898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.210944 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.211007 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.211103 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 14:56:23.211416 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.201896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.4.27.60:39001)
0812 14:56:23.202026 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.202032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.202053 (+ 21us) server_negotiation.cc:407] Connection header received
0812 14:56:23.202461 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.202468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.202512 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.202614 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.202917 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.203883 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.205441 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.205882 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.206370 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.208386 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.208393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.208393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.208399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.210096 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.210478 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.210481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.210481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.210540 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.210722 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.210722 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.210723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.210830 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.210899 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.211053 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.211116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260812 14:56:23.211469 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.203054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.62:41785 (local address 127.0.0.1:43696)
0812 14:56:23.203187 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.203218 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.203277 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.203414 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.203417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.203742 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.203874 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.203880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.204625 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.204626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.205296 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.205299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.205691 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.205692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.205804 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.206254 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.206264 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.207861 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.209653 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.209658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.209660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.209915 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.210110 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.210111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.210111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.210228 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.210515 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.210522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.210664 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.210777 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.210916 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260812 14:56:23.211895 13368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:23.213182 13228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: Bootstrap starting.
I20260812 14:56:23.214133 13228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:23.214227 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.204328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.4.27.60:39001)
0812 14:56:23.204435 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.204437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.204490 (+ 53us) server_negotiation.cc:407] Connection header received
0812 14:56:23.204554 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.204554 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.204561 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.204623 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.205335 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.205993 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.206974 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.207304 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.207368 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.210062 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.210063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.210063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.210069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.212248 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.213317 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.213319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.213319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.213378 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.213584 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.213585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.213585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.213712 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.213856 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.214045 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.214119 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
I20260812 14:56:23.214484 13228 log.cc:824] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:23.215387 13228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: No bootstrap required, opened a new log
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:23.217262 13228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.217404 13228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd6c7644a374db9a042be6fa1599b6c, State: Initialized, Role: FOLLOWER
I20260812 14:56:23.217489 13228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.217787 13228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:23.218925 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.212110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.61:36087 (local address 127.0.0.1:47838)
0812 14:56:23.212169 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.212177 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.212210 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.212350 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.212351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.212488 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.212517 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.212518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.213294 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.213295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.213762 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.213763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.214236 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.214237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.214402 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.214766 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.214771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.215941 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.217511 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.217511 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.217512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.217709 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.217980 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.217981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.217981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.218080 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.218434 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.218435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.218476 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.218641 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.218848 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.62@KRBTEST.COM
I20260812 14:56:23.219975 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.212157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.4.27.61:36087)
0812 14:56:23.212190 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.212191 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.212199 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:56:23.212260 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.212260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.212269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.212311 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.212586 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.213217 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.213833 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.214060 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.214118 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.216012 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.216013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.216013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.216019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.217425 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.217791 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.217792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.217792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.217869 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.218159 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.218160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.218160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.218321 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.218383 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.219742 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.219875 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:56:23.220930 13368 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
I20260812 14:56:23.220503 13379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.221308 13379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: This master's current role is: FOLLOWER
W20260812 14:56:23.220096 13400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:23.221372 13400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:23.223613 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.215078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.4.27.62:41785)
0812 14:56:23.215107 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.215109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.215119 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:56:23.215190 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.215191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.215199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.215250 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.215811 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.216449 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.217369 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.217823 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.217867 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.220585 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.220587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.220587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.220594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.222193 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.222590 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.222591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.222591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.222648 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.222824 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.222825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.222825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.222940 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.223059 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.223456 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.223529 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260812 14:56:23.221829 13100 master_runner.cc:186] Error getting master registration for 127.4.27.62:41785: OK, instance_id { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" instance_seqno: 1786546582917769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:56:23.224303 13100 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.60@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.27.61@KRBTEST.COM
I20260812 14:56:23.228786 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.221126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44184)
0812 14:56:23.221161 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.221169 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.221202 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.221592 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.221593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.221730 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.221768 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.221770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.222611 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.222611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.223324 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.223327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.223997 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.223997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.224050 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.224405 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.224409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.225639 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.227842 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.227843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.227843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.228051 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.228238 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.228238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.228239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.228294 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.228513 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.228513 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.228549 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.228608 (+ 59us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.228697 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260812 14:56:23.228989 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.221450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.4.27.60:39001)
0812 14:56:23.221486 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.221488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.221495 (+ 7us) server_negotiation.cc:407] Connection header received
0812 14:56:23.221512 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.221512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.221520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.221576 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.221852 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.222531 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.223408 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.223716 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.223763 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.225720 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.225721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.225721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.225728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.227758 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.228134 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.228135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.228135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.228187 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.228344 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.228345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.228346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.228458 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.228511 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.228639 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.228746 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":18}
I20260812 14:56:23.230199 13368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: Bootstrap starting.
I20260812 14:56:23.230895 13368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:23.231165 13368 log.cc:824] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:23.231885 13368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: No bootstrap required, opened a new log
I20260812 14:56:23.232758 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.224871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.4.27.61:36087)
0812 14:56:23.224902 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.224903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.224911 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:56:23.224965 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.224965 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.224974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.225023 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.225439 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.225979 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.226641 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.226921 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.226977 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.230542 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.230543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.230543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.230549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.231828 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.232137 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.232138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.232138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.232190 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.232348 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.232349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.232349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.232446 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.232555 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.232620 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.232684 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:56:23.233724 13368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.233870 13368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f815ce5878c4abbbd1ec56ee9d86c4b, State: Initialized, Role: FOLLOWER
I20260812 14:56:23.233942 13368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.234073 4204 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 14:56:23.234211 13368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:23.234187 13403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.234272 13403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:23.234895 13042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b5a9b98763c04503ae64118e2efcea3a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708
W20260812 14:56:23.235936 13423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:23.235980 13423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:23.236022 13245 master_runner.cc:186] Error getting master registration for 127.4.27.62:41785: OK, instance_id { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" instance_seqno: 1786546582917769 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 14:56:23.238090 13245 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 14:56:23.256150 13086 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } attempt: 2
Aug 12 14:56:23 dist-test-slave-6zbq krb5kdc[12943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786546582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.27.62@KRBTEST.COM for kudu/127.4.27.60@KRBTEST.COM
I20260812 14:56:23.263305 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.256370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.27.60:39001 (local address 127.0.0.1:44186)
0812 14:56:23.256412 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0812 14:56:23.256421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0812 14:56:23.256454 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 14:56:23.256589 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 14:56:23.256590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 14:56:23.256748 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 14:56:23.256788 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.256789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.257623 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.257624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.258426 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 14:56:23.258429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.258877 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.258878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 14:56:23.258932 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.259302 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 14:56:23.259307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 14:56:23.260630 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 14:56:23.262256 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.262257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.262258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.262490 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.262691 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 14:56:23.262692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 14:56:23.262692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0812 14:56:23.262755 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 14:56:23.263011 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 14:56:23.263012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 14:56:23.263055 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 14:56:23.263119 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 14:56:23.263215 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 14:56:23.263604 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.256430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44186 (local address 127.4.27.60:39001)
0812 14:56:23.256463 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.256465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.256474 (+ 9us) server_negotiation.cc:407] Connection header received
0812 14:56:23.256512 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.256513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.256521 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.256571 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.256857 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.257548 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.258500 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.258808 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.258874 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.260722 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.260724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.260724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.260730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.262183 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.262556 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.262557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.262558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.262611 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.262822 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.262824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.262824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.262949 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.263010 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.263456 (+ 446us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.263526 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260812 14:56:23.264922 13086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a: Bootstrap starting.
I20260812 14:56:23.265559 13086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a: Neither blocks nor log segments found. Creating new log.
I20260812 14:56:23.265836 13086 log.cc:824] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a: Log is configured to *not* fsync() on all Append() calls
I20260812 14:56:23.266646 13086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a: No bootstrap required, opened a new log
I20260812 14:56:23.267931 13086 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.268046 13086 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a9b98763c04503ae64118e2efcea3a, State: Initialized, Role: FOLLOWER
I20260812 14:56:23.268112 13086 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.268378 13086 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 14:56:23.268343 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.268426 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:23.269737 13042 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 14:56:23.269845 13042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 0 FOLLOWER]: Advancing to term 1
W20260812 14:56:23.270020 13453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 14:56:23.270077 13453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 14:56:23.270684 13042 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.271190 13181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5a9b98763c04503ae64118e2efcea3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0bd6c7644a374db9a042be6fa1599b6c"
I20260812 14:56:23.271323 13181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:23.272107 13181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a9b98763c04503ae64118e2efcea3a in term 1.
I20260812 14:56:23.272311 12999 leader_election.cc:302] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd6c7644a374db9a042be6fa1599b6c, b5a9b98763c04503ae64118e2efcea3a; no voters:
I20260812 14:56:23.272418 13427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 14:56:23.272537 13427 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [term 1 LEADER]: Becoming Leader. State: Replica: b5a9b98763c04503ae64118e2efcea3a, State: Running, Role: LEADER
I20260812 14:56:23.272609 13427 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } }
I20260812 14:56:23.272980 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a9b98763c04503ae64118e2efcea3a. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.273043 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:23.273315 13042 leader_election.cc:288] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [CANDIDATE]: Term 1 election: Requested vote from peers 0bd6c7644a374db9a042be6fa1599b6c (127.4.27.61:36087), 1f815ce5878c4abbbd1ec56ee9d86c4b (127.4.27.60:39001)
I20260812 14:56:23.273546 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5a9b98763c04503ae64118e2efcea3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b"
I20260812 14:56:23.273655 13323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 14:56:23.275295 13323 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a9b98763c04503ae64118e2efcea3a in term 1.
I20260812 14:56:23.279556 13460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 14:56:23.279767 13460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 14:56:23.283547 13323 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [term 1 FOLLOWER]: Refusing update from remote peer b5a9b98763c04503ae64118e2efcea3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:56:23.283669 13181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [term 1 FOLLOWER]: Refusing update from remote peer b5a9b98763c04503ae64118e2efcea3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 14:56:23.283841 13456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:56:23.283996 13456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 14:56:23.284958 13379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a9b98763c04503ae64118e2efcea3a. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.285036 13379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:23.285648 13403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5a9b98763c04503ae64118e2efcea3a. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.285732 13403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:23.293561 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.278572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.4.27.60:39001)
0812 14:56:23.278612 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.278614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.282511 (+ 3897us) server_negotiation.cc:407] Connection header received
0812 14:56:23.282570 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.282571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.282580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.282635 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.286839 (+ 4204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.287492 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.288724 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.289020 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.289057 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.290403 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.290404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.290404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.290411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.292100 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.292722 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.292723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.292724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.292796 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.293008 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.293010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.293010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.293129 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.293209 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.293410 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.293480 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:56:23.294353 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.280003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.4.27.61:36087)
0812 14:56:23.280037 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.280038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.282765 (+ 2727us) server_negotiation.cc:407] Connection header received
0812 14:56:23.282795 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.282795 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.282804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.282844 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.285908 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.286673 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.287794 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.288091 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.288139 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.291318 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.291319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.291320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.291326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.293033 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.293410 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.293411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.293411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.293462 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.293699 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.293700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.293701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.293818 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.293880 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.294182 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.294252 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:56:23.298511 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:23.278877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.4.27.62:41785)
0812 14:56:23.278921 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.278923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.282668 (+ 3745us) server_negotiation.cc:407] Connection header received
0812 14:56:23.282687 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.282688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.282696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.282739 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.289634 (+ 6895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.290336 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.291590 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:23.291858 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:23.291896 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:23.295907 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 14:56:23.295908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 14:56:23.295908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 14:56:23.295915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0812 14:56:23.297361 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.297731 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.297731 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.297732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.297797 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 14:56:23.297975 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 14:56:23.297976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 14:56:23.297976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0812 14:56:23.298085 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 14:56:23.298150 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:23.298326 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 14:56:23.298430 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260812 14:56:23.300114 13427 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.300192 13427 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:23.300297 13403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.300379 13403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:23.300366 13472 mvcc.cc:205] Tried to move back new op lower bound from 7317694805127188480 to 7317694805085261824. Current Snapshot: MvccSnapshot[applied={T|T < 7317694805127188480}]
I20260812 14:56:23.300567 13481 mvcc.cc:205] Tried to move back new op lower bound from 7317694805127188480 to 7317694805085261824. Current Snapshot: MvccSnapshot[applied={T|T < 7317694805127188480}]
I20260812 14:56:23.304168 13379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.304248 13379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 14:56:23.303678 13456 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5a9b98763c04503ae64118e2efcea3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a9b98763c04503ae64118e2efcea3a" member_type: VOTER last_known_addr { host: "127.4.27.62" port: 41785 } } peers { permanent_uuid: "0bd6c7644a374db9a042be6fa1599b6c" member_type: VOTER last_known_addr { host: "127.4.27.61" port: 36087 } } peers { permanent_uuid: "1f815ce5878c4abbbd1ec56ee9d86c4b" member_type: VOTER last_known_addr { host: "127.4.27.60" port: 39001 } } }
I20260812 14:56:23.304497 13456 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a [sys.catalog]: This master's current role is: LEADER
I20260812 14:56:23.303943 13460 catalog_manager.cc:1347] Generated new cluster ID: 2d894badfe044c75a1efeae3c8dc1aca
I20260812 14:56:23.304560 13460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 14:56:23.310071 13475 mvcc.cc:205] Tried to move back new op lower bound from 7317694805127188480 to 7317694805085261824. Current Snapshot: MvccSnapshot[applied={T|T < 7317694805127188480 or (T in {7317694805127188480})}]
I20260812 14:56:23.310537 13460 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 14:56:23.311187 13460 catalog_manager.cc:1504] Loading token signing keys...
I20260812 14:56:23.317979 13460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5a9b98763c04503ae64118e2efcea3a: Generated new TSK 0
I20260812 14:56:23.318151 13460 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 14:56:23.342911 13376 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.341540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44198 (local address 127.4.27.60:39001)
0812 14:56:23.341584 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.341586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.342238 (+ 652us) server_negotiation.cc:407] Connection header received
0812 14:56:23.342353 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.342354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.342366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.342397 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.342842 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44198: BlockingRecv error: recv got EOF from 127.0.0.1:44198 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260812 14:56:23.343812 13095 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.341952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.4.27.62:41785)
0812 14:56:23.341988 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.341990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.343309 (+ 1319us) server_negotiation.cc:407] Connection header received
0812 14:56:23.343398 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.343399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.343409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.343436 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.343738 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43724: BlockingRecv error: recv got EOF from 127.0.0.1:43724 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:56:23.345002 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.344259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.4.27.61:36087)
0812 14:56:23.344295 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.344297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.344457 (+ 160us) server_negotiation.cc:407] Connection header received
0812 14:56:23.344549 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.344550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.344560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.344587 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.344945 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47876: BlockingRecv error: recv got EOF from 127.0.0.1:47876 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:56:23.347290 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.346586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.4.27.61:36087)
0812 14:56:23.346622 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.346624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.346799 (+ 175us) server_negotiation.cc:407] Connection header received
0812 14:56:23.346901 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.346902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.346922 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.346949 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.347251 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47880: BlockingRecv error: recv got EOF from 127.0.0.1:47880 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260812 14:56:23.350539 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.349690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.4.27.61:36087)
0812 14:56:23.349742 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.349744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.349960 (+ 216us) server_negotiation.cc:407] Connection header received
0812 14:56:23.350051 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.350052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.350062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.350089 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.350501 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47882: BlockingRecv error: recv got EOF from 127.0.0.1:47882 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260812 14:56:23.354372 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.353657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.4.27.61:36087)
0812 14:56:23.353689 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.353690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.353895 (+ 205us) server_negotiation.cc:407] Connection header received
0812 14:56:23.353980 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.353981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.353990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.354016 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.354309 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47894: BlockingRecv error: recv got EOF from 127.0.0.1:47894 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260812 14:56:23.360260 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.359519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.4.27.61:36087)
0812 14:56:23.359562 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.359564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.359781 (+ 217us) server_negotiation.cc:407] Connection header received
0812 14:56:23.359873 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.359874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.359883 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.359909 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.360218 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47904: BlockingRecv error: recv got EOF from 127.0.0.1:47904 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260812 14:56:23.370260 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.369497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.4.27.61:36087)
0812 14:56:23.369530 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.369531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.369811 (+ 280us) server_negotiation.cc:407] Connection header received
0812 14:56:23.369900 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.369900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.369910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.369935 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.370219 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47912: BlockingRecv error: recv got EOF from 127.0.0.1:47912 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260812 14:56:23.388027 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.387444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.4.27.61:36087)
0812 14:56:23.387482 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.387483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.387587 (+ 104us) server_negotiation.cc:407] Connection header received
0812 14:56:23.387635 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.387635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.387646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.387691 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.387982 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47916: BlockingRecv error: recv got EOF from 127.0.0.1:47916 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260812 14:56:23.421713 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.421038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.4.27.61:36087)
0812 14:56:23.421115 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.421117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.421159 (+ 42us) server_negotiation.cc:407] Connection header received
0812 14:56:23.421231 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.421232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.421245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.421316 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.421653 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47928: BlockingRecv error: recv got EOF from 127.0.0.1:47928 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260812 14:56:23.487677 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.486912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.4.27.61:36087)
0812 14:56:23.486990 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.486992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.487106 (+ 114us) server_negotiation.cc:407] Connection header received
0812 14:56:23.487170 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.487171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.487184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.487236 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.487619 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47932: BlockingRecv error: recv got EOF from 127.0.0.1:47932 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 14:56:23.617430 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.616800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.4.27.61:36087)
0812 14:56:23.616863 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.616864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.616943 (+ 79us) server_negotiation.cc:407] Connection header received
0812 14:56:23.616997 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.616998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.617007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.617057 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.617370 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47934: BlockingRecv error: recv got EOF from 127.0.0.1:47934 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260812 14:56:23.875139 13235 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 14:56:23.874573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.4.27.61:36087)
0812 14:56:23.874647 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:23.874649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:23.874659 (+ 10us) server_negotiation.cc:407] Connection header received
0812 14:56:23.874715 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:23.874716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:23.874727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:23.874776 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 14:56:23.875078 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47946: BlockingRecv error: recv got EOF from 127.0.0.1:47946 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 14:56:24.221855 13400 catalog_manager.cc:1259] Loaded cluster ID: 2d894badfe044c75a1efeae3c8dc1aca
I20260812 14:56:24.221938 13400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: loading cluster ID for follower catalog manager: success
I20260812 14:56:24.223100 13400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: acquiring CA information for follower catalog manager: success
I20260812 14:56:24.223421 13400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bd6c7644a374db9a042be6fa1599b6c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:56:24.236361 13423 catalog_manager.cc:1259] Loaded cluster ID: 2d894badfe044c75a1efeae3c8dc1aca
I20260812 14:56:24.236411 13423 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: loading cluster ID for follower catalog manager: success
I20260812 14:56:24.237159 13423 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: acquiring CA information for follower catalog manager: success
I20260812 14:56:24.237417 13423 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f815ce5878c4abbbd1ec56ee9d86c4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 14:56:24.390535 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:24.388266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.4.27.61:36087)
0812 14:56:24.388367 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:24.388368 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:24.388380 (+ 12us) server_negotiation.cc:407] Connection header received
0812 14:56:24.388460 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:24.388461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:24.388511 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:24.388576 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:24.388704 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.389184 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.389879 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.390060 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.390102 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:24.390208 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:24.390313 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:24.390402 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:24.390438 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 14:56:24.390454 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
I20260812 14:56:24.394111 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:24.392172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.4.27.60:39001)
0812 14:56:24.392288 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:24.392290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:24.392298 (+ 8us) server_negotiation.cc:407] Connection header received
0812 14:56:24.392332 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:24.392333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:24.392364 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:24.392425 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:24.392499 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.392949 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.393572 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.393753 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.393794 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:24.393884 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:24.393966 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:24.394011 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:24.394039 (+ 28us) server_negotiation.cc:299] Negotiation successful
0812 14:56:24.394055 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260812 14:56:24.398152 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:24.395603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.4.27.61:36087)
0812 14:56:24.395638 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:24.395639 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:24.395740 (+ 101us) server_negotiation.cc:407] Connection header received
0812 14:56:24.395842 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:24.395843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:24.395852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:24.395880 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:24.396026 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.396509 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.397390 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.397669 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.397731 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:24.397847 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:24.397876 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:24.397919 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:24.397969 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 14:56:24.397986 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260812 14:56:24.398433 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:24.395549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.4.27.60:39001)
0812 14:56:24.395695 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:24.395696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:24.395816 (+ 120us) server_negotiation.cc:407] Connection header received
0812 14:56:24.395891 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:24.395892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:24.395900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:24.395951 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:24.396082 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.396779 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.397668 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.397945 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.398001 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:24.398162 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:24.398193 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:24.398237 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:24.398330 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 14:56:24.398370 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 14:56:24.398900 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 14:56:24.395507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.4.27.62:41785)
0812 14:56:24.395717 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 14:56:24.395719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0812 14:56:24.395786 (+ 67us) server_negotiation.cc:407] Connection header received
0812 14:56:24.395869 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 14:56:24.395870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 14:56:24.395919 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 14:56:24.395984 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 14:56:24.396141 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.396872 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.397707 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 14:56:24.397987 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 14:56:24.398042 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 14:56:24.398464 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 14:56:24.398647 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 14:56:24.398716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 14:56:24.398788 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 14:56:24.398816 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":50,"threads_started":1}
I20260812 14:56:24.400697 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 12968
I20260812 14:56:24.424443 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13100
I20260812 14:56:24.433233 4204 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktWKW5U/build/release/bin/kudu with pid 13245
2026-08-12T14:56:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2909 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5831 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270637 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST